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1.0  SUMMARY 


Recent  advances  in  Integrated  Computational  Materials  Engineering  (ICME),  and  manufacturing 
process  models,  along  with  increased  use  of  probabilistic  methods  and  uncertainty  quantification 
can  be  applied  to  accelerate  maturation  of  new  manufacturing  processes  and  technology. 
Combined  with  new  processes  to  identify,  assess,  and  address  technology  risks  during 
development,  it  is  believed  that  significant  acceleration  in  technology  maturation  can  be 
achieved.  Rapid  maturation  and  qualification  would  enable  more  effective  transition  of 
technology  into  production  use  for  new  systems,  enable  faster,  more  capable  response  to  design 
changes,  and  reduce  risk  of  insertion  of  new  materials  and  manufacturing  processes.  These  goals 
are  consistent  with,  and  supportive  of,  Air  Force  Research  Laboratories  (AFRL)  strategic 
initiatives,  including  Moving  Manufacturing  Left,  Agile  Manufacturing,  and  Digital  Thread. 

AFRL  desires  to  develop  a  recommended  process  and  associated  guideline  document  supporting 
more  rapid  maturation  and  risk  reduction  for  new  Manufacturing  Technologies  (MT),  including 
the  recommended  application  of  promising  tools  and  methods.  This  effort  was  expected  to  be 
especially  relevant  to  future  “quick  reaction  systems,”  where  total  development  time  may  be 
reduced,  and  also  to  future  systems  where  total  unit  production  may  be  relatively  low.  To  this 
end,  the  Manufacturing  Technology  Directorate  at  the  Air  Force  Research  Laboratories 
(AFRL/RXM)  sponsored  a  six  month  “Phase  0”  study  task  to  make  an  initial  assessment  of 
promising  tools  and  methods,  formulate  a  high-level  technical  strategy,  and  generate  an  initial 
outline  of  a  guideline  document  supporting  the  rapid  maturation  and  risk  reduction  for  new 
manufacturing  technologies. 

The  study  task  identified  candidate  topic  areas  for  further  evaluation,  and  then  proposed  an 
approach  to  integrate  these  activities  with  the  current  MRL/MRA  process.  The  proposed 
approach  advocates  an  additional  Manufacturing  Thread  in  the  MRL  process:  “ Systems 
Engineering  for  Manufacturing  Technology.  ”  This  proposed  thread  consists  of  six  integrated 
sub-threads,  comprising  the  promising  tools  and  methods  that  were  identified  and  evaluated 
under  the  study  task.  Alternatively,  the  proposed  activities  could  be  developed  into  a  “best 
practice,”  a  recommended  procedure,  or  an  engineering  work  instruction.  Proposed  activities 
were  defined,  sequenced,  and  aligned  with  MRL  gates.  It  is  thought  that  these  activities,  when 
employed  in  an  integrated  manner,  could  effectively  accelerate  the  maturation  of  new 
Manufacturing  Technologies,  and  reduce  the  risk  associated  with  implementation.  This  report 
summarizes  results  of  the  study  task. 
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2.0  INTRODUCTION 


Development,  certification  and  sustainment  of  Department  of  Defense  (DoD)  weapon  systems 
are  lengthy,  expensive  processes,  which  often  require  iterations  in  design  configuration  or 
changes  in  materials  or  manufacturing  processes  during  development.  Development  of  new 
technology  is  not  always  in  sync  with  schedule  requirements  for  system  acquisition,  inhibiting 
insertion.  This  is  especially  true  for  new  manufacturing  processes  and  technology,  where  lack  of 
prior  production  experience  increases  risk  and  inhibits  commitment  to  new  processes  and 
technology,  resulting  in  significant  missed  opportunity.  Future  DoD  requirements  may 
exacerbate  these  issues.  Demand  for  “quick  reaction  systems”  -  where  intent  is  to  rapidly 
develop  new  systems,  as  well  as  systems  specifically  designated  for  “accelerated  acquisition,” 
and  the  likelihood  that  many  new  systems  will  have  limited  unit  production  requirements,  will  all 
challenge  the  legacy  technology  development  and  implementation  processes. 

Recent  advances  in  Integrated  Computational  Materials  Engineering  (ICME),  in  manufacturing 
process  models,  and  increased  use  of  probabilistic  methods  and  uncertainty  quantification  may 
enable  accelerated  maturation  of  new  manufacturing  processes  and  technology.  Combined  with 
new  processes  to  identify,  assess,  and  address  technology  risks  during  development,  it  is  believed 
that  significant  acceleration  in  technology  maturation  can  be  achieved.  Rapid  maturation  and 
qualification  would  enable  more  effective  transition  of  technology  into  production  use  for  new 
systems,  enable  faster,  more  capable  response  to  design  changes,  and  reduce  risk  of  insertion  of 
new  materials  and  manufacturing  processes.  These  goals  are  believed  to  be  consistent  with,  and 
supportive  of,  DoD  and  AFRL  strategic  initiatives,  including  Engineered  Resilient  Systems, 
Moving  Manufacturing  Left,  Agile  Manufacturing,  and  Digital  Thread. 

AFRL  desires  to  develop  a  recommended  process  and  associated  guideline  document  supporting 
more  rapid  maturation  and  risk  reduction  for  new  manufacturing  technologies  (MT),  including 
the  recommended  application  of  promising  tools  and  methods.  To  this  end,  AFRL  sponsored  a 
study  task  to  make  an  initial  assessment  of  promising  tools  and  methods,  formulate  a  high-level 
technical  strategy,  and  generate  an  initial  outline  of  a  guideline  document  supporting  the  rapid 
maturation  and  risk  reduction  for  new  manufacturing  technologies  (MT).  The  study  effort  was 
organized  into  three  tasks: 

•  Task  1:  Analysis  of  Background  Information 

•  Task  2:  Identification  of  Methods  to  Accelerate  MT  Maturation  and  Risk  Reduction 

•  Task  3:  Formulation  of  an  Initial  Guideline  Process  and  Associated  Research  Plan 

The  objective  of  Task  1  was  to  review  methods  and  guidelines  that  the  Department  of  Defense 
(DoD)  currently  employs  to  assess  MT  development  and  maturation.  These  methods  and 
guidelines  included  the  Defense  Acquisition  Guidebook  (DAG),  and  DoD  Instruction  5000.02 
for  highest  level  context,  and  Manufacturing  Readiness  Levels  (MRLs)  and  associated 
Manufacturing  Readiness  Assessments  (MRAs)  for  practical,  application  level  guidelines.  In 
addition,  relevant  team  processes  available  in  the  open  literature  for  risk  assessment  and 
mitigation,  including  “expert  elicitation”  and  subject  matter  expert  (SME)  review  processes  were 
assessed  for  relevance.  Key  issues  and  challenges  associated  with  the  rapid  maturation  and  risk 
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reduction  of  new  manufacturing  processes  were  also  reviewed  in  the  context  of  a  few  selected 
prior  MT  programs.  The  intent  was  to  frame  and  focus  efforts  planned  in  Tasks  2  and  3  of  this 
study. 

Several  candidate  topics  were  identified  for  further  consideration  and  assessment  under  the  study 
task: 

•  Early  risk  identification  and  mitigation  approaches 

•  Increased  use  of  computational  modeling  for  materials  and  processes  (ICME),  including 
empirically  based  modeling 

•  Verification  and  Validation  methods  for  such  modeling 

•  Uncertainty  Quantification  and  probabilistic  assessments 

•  Feedback  and  updating  methods  for  refining  models  based  on  early  data 

•  Increased  use  of  in-process  monitoring  and  advanced  inspection  methods  for  process 
monitoring  and  control 

•  Definition  and  execution  of  critical  experiments. 

The  candidate  topics  were  evaluated  and  refined,  in  the  context  of  the  current  MRL/MRA 
process  and  the  associated  Manufacturing  Threads  that  have  been  defined.  It  should  be  noted  that 
in  several  cases,  the  recommended  activities  list  associated  with  the  “manufacturing  threads” 
already  contain  or  infer  some  of  the  items  identified  for  further  assessment  under  this  study  task, 
although  in  current  practice  these  items  are  generally  not  well  defined  and  occur  later  in  the 
process  than  would  be  most  effective. 

One  key  conclusion  was  that  these  topics  -  both  strategies  and  methods  -  would  be  most 
effective  when  employed  early  in  an  integrated,  iterative,  and  collective  manner.  They  would  not 
be  nearly  as  effective  if  applied  individually  or  independently.  A  conceptual  framework  was 
developed  to  illustrate  where  in  the  MRL/MRA  process  these  methods  should  be  applied. 

The  current  MRL/MRA  process  is  well  defined  and  accepted;  it  is  also  comprehensive  in  both 
scope  and  application.  When  properly  practiced,  it  effectively  assesses  and  reduces 
implementation  risks  for  new  manufacturing  technologies.  The  most  significant  challenge  for 
this  Study  Task  was  to  identify,  define,  and  recommend  processes  and/or  methods  that  could 
effectively  accelerate  maturation  of  manufacturing  technologies,  and  to  provide  confidence  to 
key  decision-makers  that  such  technologies  could  be  successfiilly  implemented,  with  an 
acceptable  level  of  risk. 

The  identified  topic  areas  were  incorporated  in  Task  3  efforts:  Formulation  of  an  Initial 
Guideline  Process  and  Associated  Research  Plan.  The  objectives  of  Task  3  were  to  refine  and 
expand  upon  the  candidate  topics  previously  identified,  and  to  outline  an  initial  guideline  process 
for  implementation.  The  approach  was  to  identify  recommended  activities,  supporting  methods, 
tools  and  references,  and  relate  these  to  the  current  MRL  process.  Feedback  from  several  AFRL 
and  industry  experts  was  also  received  and  incorporated.  The  results  were  used  to  generate  a 
suite  of  recommended  activities,  which  were  consolidated  into  a  framework  and  integrated  with 
the  current  MRL/MRA  process.  In  addition,  a  proposed  set  of  questions  was  developed  to  assist 
assessment  of  when  such  activities  would  be  appropriate.  The  result  is  a  recommendation  that 
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one  additional  Manufacturing  Thread  be  considered  for  inclusion  in  the  MRL/MRA 
framework:  “Systems  Engineering  for  Manufacturing  Technology  (SEMT)  ”  This 
recommended  additional  Manufacturing  Thread  consists  of  six  (6)  sub-threads,  which  contain  the 
recommended  activities  and  the  proposed  linkage  to  the  MRL/MRA  process.  The 
recommendations  are  believed  compatible  and  aligned  with  the  current  process.  Alternatively , 
the  recommended  activities  under  SEMT  could  readily  be  adapted  as  a  “Recommended  Best 
Practice,  ”  or  as  a  program  or  engineering  work  instruction,  as  the  proposed  activities  are 
aligned  and  entirely  consistent  with  the  current  MRL/MRA  process. 
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3.0  METHODS,  ASSUMPTIONS,  AND  PROCEDURES 
3.1  General  Approach 

The  objectives  of  the  study  task  were  to  develop  a  recommended  process  and  associated 
guideline  document  supporting  more  rapid  maturation  and  risk  reduction  for  new  manufacturing 
technologies  (MT),  including  the  recommended  application  of  promising  tools  and  methods. 
These  tools  and  methods  were  to  include  increased  use  of  computational  models,  the  associated 
verification  and  validation  processes  needed  to  ensure  confidence  in  such  models,  increased  use 
of  uncertainty  quantification  and  probabilistic  methods,  and  increased  development  and  use  of 
in-process  monitoring  and  control.  In  addition,  the  study  task  was  intended  to  explore  methods 
and  processes  to  identify  technology  and  development  risks  early,  and  to  develop  risk  mitigation 
plans  as  appropriate.  The  general  approach  that  was  taken  consisted  of  three  sequential  tasks: 

•  Analysis  of  Background  Information 

•  Identification  of  Methods  to  Accelerate  MT  Maturation  and  Risk  Reduction 

•  Formulation  of  an  Initial  Guideline  Process  and  Associated  Research  Plan 

Under  the  first  task,  current  DoD  and  USAF  processes  for  system  acquisition  and  manufacturing 
technology  development  were  reviewed.  These  included  the  Defense  Acquisition  Guidebook 
(DAG),  and  DoD  Instruction  5000.02  for  highest  level  context,  and  Manufacturing  Readiness 
Levels  (MRLs)  and  associated  Manufacturing  Readiness  Assessments  (MRAs)  for  practical, 
application  level  guidelines. 

Under  the  second  task,  candidate  tools  and  methods  were  identified  and  evaluated  that  were 
believed  capable  of  accelerating  the  maturation  of  new  manufacturing  technologies,  and  reducing 
risk  of  insertion.  The  resulting  topics,  which  were  listed  in  the  Introduction  section,  above,  were 
reviewed  in  some  detail.  Descriptions  of  activities  and  rationale  for  use  were  generated,  and 
literature  references  were  provided  as  appropriate. 

The  approach  taken  under  the  third  task  was  to  consolidate  the  activities  associated  with  these 
topics  into  an  integrated  suite  of  recommended  activities,  in  a  manner  that  would  be  fully  aligned 
and  consistent  with  the  current  MRL/MRA  process.  Specific  recommended  activities  were 
defined,  sequenced,  and  aligned  with  recommended  MRL  gates.  The  result  is  a  recommendation 
that  one  additional  Manufacturing  Thread  be  considered  for  inclusion  in  the  MRL/MRA 
framework:  “Systems  Engineering  for  Manufacturing  Technology  (SEMT)”  This  recommended 
additional  Manufacturing  Thread  consists  of  six  (6)  sub-threads,  which  contain  the  recommended 
activities  and  the  proposed  linkage  to  the  MRL/MRA  process.  The  recommendations  are 
believed  compatible  and  aligned  with  the  current  process.  Alternatively,  the  recommended 
activities  under  SEMT  could  readily  be  adapted  as  a  “Recommended  Best  Practice”  or  an 
engineering  instruction  or  guideline,  as  the  proposed  activities  are  aligned  and  entirely 
consistent  with  the  current  MRL/MRA  process. 


5 

Distribution  Statement  A.  Approved  for  public  release  (PA);  distribution  unlimited. 


3.2  High  Level  Context  and  the  Current  MRL/MRA  Process 

The  approach  selected  was  to  review  current  guidelines  and  practices  used  for  assessment  and 
decision-making  relative  to  materials  and  manufacturing  technologies,  specifically  for  those 
associated  with  structural  applications  in  aerospace  systems.  With  the  intent  to  ensure  proper 
context,  the  starting  point  for  this  review  was  the  general  acquisition  process  used  by  the 
Department  of  Defense  (DoD)  for  weapons  systems.  The  acquisition  process  for  such  an 
enormous  breadth  of  applications  is,  of  course,  extraordinarily  large  and  complex.  It  was  also 
recognized  that  these  processes  are  also  intended  for  application  at  the  system  level,  and  most 
often  for  major  weapons  systems  or  acquisitions.  The  latest  guidelines,  which  are  summarized  in 
the  Defense  Acquisition  Guidebook  [1],  provides  guidance  and  reference  paths  for  many  key 
activities  and  decision  processes  which  are  generically  applicable  to  DoD  weapons  system 
acquisition.  It  addresses  both  the  pre-acquisition  and  acquisition  phases  in  the  weapons  system 
life  cycle.  Relevant  reference  material  for  this  study  task  was  found  in  Chapter  4  -  Systems 
Engineering.  Specifically,  within  the  Milestone  A  and  Technology  Development  phase  write-up 
in  Section  4.2.4,  use  of  Technology  Readiness  Assessment  (TRA)  was  cited.  Later,  in  Section 
4.2.5  pertaining  to  Milestone  B  and  the  Engineering  and  Manufacturing  Development  phase 
(EMD),  the  MRL  Deskbook  and  use  of  Manufacturing  Readiness  Levels  (MRL)  was  cited  as 
“one  source”  for  how  to  assess  manufacturing  and  production  readiness.  The  references  in  the 
Defense  Acquisition  Guidelines  were  not  at  all  specific  in  terms  of  methods  and  processes  that 
were  recommended  for  technology  or  manufacturing  readiness  assessment  or  management. 

The  DAG  guidelines  do,  however,  reference  the  DoD  instruction  DODI  5000.02  [2]  for  detailed 
requirements,  specific  statutory  requirements,  and  connectivity  between  Defense  Acquisition 
Milestones  (eg:  Milestones  A,  B,  and  C),  program  needs,  and  the  assessment  and  management 
of  program  technical  risks.  The  generic  sequence  of  program  phases,  key  milestones,  and 
associated  supporting  milestones  are  shown  in  Figure  1,  as  extracted  directly  from  DODI 
5000.02. 
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Figure  1:  Defense  Acquisition  Process  Sequence  and  Milestones  from  DODI5000.02 

(Dec  2008) 
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Technology  and  Manufacturing  Readiness  Levels  have  been  linked  to  the  acquisition  process  and 
milestones  shown  in  Figure  1.  In  general,  it  has  been  interpreted  that  both  Technology  and 
Manufacturing  Readiness  Levels  of  6  are  required  for  a  new  technology  or  manufacturing 
process  prior  to  Defense  Acquisition  Milestone  B,  which  marks  the  beginning  of  the  Engineering 
and  Manufacturing  Development  phase  (EMD),  where  detailed  design  commences.  In  addition, 
TRL  and  MRL-4  has  been  linked  to  Milestone  A,  the  beginning  of  the  Technology  Development 
Phase,  and  MRL-8  with  Milestone  C,  the  beginning  of  the  Production  and  Deployment  Phase, 
where  Low  Rate  Initial  Production  (LRIP)  is  initiated  for  major  weapons  systems.  The  TRL  and 
MRL  gates  are  shown  in  relation  to  the  Defense  Acquisition  System  milestones  in  Figure  2, 
below. 
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Figure  2:  MRL  and  TRL  Gates  in  Relation  to  Acquisition  Phases  and  Milestones 


In  actual  practice,  many  if  not  all  programs  have  one  or  more  technologies  that  are  not 
considered  “mature”  (TRL/MRL-6  or  greater)  at  commencement  of  Milestone  B.  A  2006  study 
by  the  GAO  [3]  found  that  this  was  the  case  for  a  majority  of  major  weapons  systems:  that 
immature  technologies  were  related  to  future  cost  and  schedule  growth,  and  that  if  a  program 
was  launched  with  “immature”  technologies,  these  issues  would  persist.  GAO  advocated 
utilization  of  “knowledge  based”  decision  points  for  assessing  both  technologies  (at  Milestone  B) 
and  production  readiness  (at  Milestone  C).  GAO  issued  many  related  studies  in  subsequent 
years,  some  of  which  specifically  recognize  and  recommend  use  of  the  TRL  and  MRL  processes. 
One  of  particular  interest  is  GAO- 10-439  [4],  which  contains  specific  recommendations  that 
DoD  adopt  wide  use  of  MRLs:  “GAO  recommends  that  the  Secretary  of  Defense  require  the  use 
of  MRLs  across  DOD  programs,  strengthen  the  MRL  criteria  (process  control)  for  production 
start,  assess  the  need  for  tools,  and  assess  the  manufacturing  workforce  to  address  knowledge 
gaps.  ” 


Very  recently,  Deputy  Secretary  of  Defense  Ashton  B.  Carter  cancelled  DODI  5000.02  in  its 
entirety  (excepting  Enclosure  9)  and  then  replaced  it  with  Interim  DODI  5000.02  [5],  The  new 
interim  document  contained  some  significant  changes,  which  are  described  briefly  here  due  to 
their  perceived  relevance  to  the  goals  and  approach  of  this  study  task.  Specifically,  Interim 
DODI  5000.02  describes  four  basic  models  as  well  as  hybrid  combinations  for  the  acquisition 
process  and  sequence,  as  compared  with  a  single  model  shown  in  the  previous  version  of  DODI 
5000.02  (see  Figure  1,  above).  The  apparent  intent  is  to  increase  flexibility  in  the  acquisition 
process  and  tailor  requirements  to  specific  program  needs,  as  permitted  within  statutory 
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requirements.  Two  of  the  four  models  are  regarded  especially  relevant  to  the  study  task:  Model 
1:  Hardware  Intensive  Program,  and  Model  4:  Accelerated  Acquisition  Program.  The  remaining 
models  were  for  software  intensive  programs  and  hybrid  combinations. 


The  graphical  depiction  of  Model  1:  Hardware  Intensive  Program  is  shown  in  Figure  3,  below. 
Comparison  with  the  acquisition  process  management  chart  previously  presented  in  Figure  1 
shows  significant  differences  in  that  the  transition  line  between  phases  is  no  longer  clearly 
defined,  and  related  activity  is  shown  before  and  after  key  milestones  B  and  C,  as  well  as  in 
transition  from  low  rate  to  full  production. 
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Figure  3:  Defense  Acquisition  Process  Sequence  and  Milestones  for  “Model  1  - 
Hardware  Intensive  Program,”  from  Interim  DODI  5000.02  (Nov  2013) 


The  graphical  depiction  of  Model  4:  Accelerated  Acquisition  Program  is  presented  in  Figure  4.  It 
shows  even  more  striking  changes  from  Figure  1:  Milestones  A  and  B  are  essentially  concurrent, 
and  it  shows  that  technology  and  manufacturing  process  development  will  occur  concurrently 
with  Engineering  and  Manufacturing  Development. 


Programs  following  this  acquisition  management  model  will  undoubtedly  require  additional 
focus  on  assessment  and  mitigation  of  program  risks  posed  by  technology  and  manufacturing 
readiness. 


This  summary  of  methods  and  guidelines  that  DoD  currently  employs  to  manage  weapons 
system  development  and  acquisition  was  clearly  at  a  high-level  -  certainly  at  the  systems  level, 
and  generally  most  applicable  for  major  systems.  The  recently  defined  (or  modified)  acquisition 
models  are  regarded  especially  relevant  to  provide  guidance  to  the  study  task,  since  the  goal  of 
the  current  study  task  is  to  identify  potential  tools  and  methods  to  accelerate  manufacturing 
technology  maturation,  and  to  reduce  program  risk  of  implementation.  It  is  not  known  at  this 
time  whether  (or  exactly  how)  the  changes  contained  in  Interim  DODI  5000.02  will  affect  the 
application  and  use  of  the  TRL  and  MRL  processes  in  the  future.  However,  the  objectives  of  this 
study  task  seem  to  be  well-aligned  to  support  the  changes  in  Interim  DODI  5000.02. 
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Figure  4:  Defense  Acquisition  Process  Sequence  and  Milestones  for  “Model  4  - 
Accelerated  Acquisition  Program,”  from  Interim  DODI  5000.02  (Nov  2013) 

A  key  objective  for  the  study  task  was  to  identify  and  evaluate  methods  or  processes  that  might 
be  used  to  accelerate  maturation  of  manufacturing  technologies  and  reduce  program  risk  of 
implementation.  The  approach  was  to  do  this  within  the  construct  of  the  current  MRL  and  MRA 
process.  A  brief  review  of  the  current  MRL  process  is  presented  in  the  following  section. 

3.3  Review  of  MRL  and  MRA  Process 

Overview  of  Key  MRL/MRA  Elements:  In  the  1970’s,  NASA  developed  the  concept  of 
technology  readiness  levels,  as  an  objective  method  to  assess  current  readiness  for  specific 
applications.  NASA  subsequently  published  Definitions  of  the  Technology  Readiness  Levels 
(TRLs)  and  associated  gate  criteria  in  the  late  1980s  [6]  to  guide  technology  development  for 
future  space  systems.  The  TRL  process,  and  the  associated  Technology  Readiness  Assessment 
process  (TRA),  are  used  extensively  today  by  both  government  and  industry  -  either  directly  or 
with  some  modifications.  There  are  nine  TRL  levels,  although  various  organizations,  notably 
AFRL,  have  refined  the  gate  criteria  and  developed  comprehensive  tools  to  help  objectively 
assess  TRL  status.  The  current  maturity  of  a  technology  is  now  often  communicated  by  simply 
stating  its  TRL  level.  It  should  be  noted  that  the  TRL  of  a  technology  is  evaluated  in  the  context 
of  a  specific  application,  usually  for  a  specific  product  or  system. 

One  weakness  in  the  TRL  process  (at  least  as  commonly  applied)  was  its  lack  of  focus  on 
manufacturing  and  production  readiness  for  new  technologies,  especially  for  new  materials  or 
new  manufacturing  processes.  This  was  recognized,  and  in  2001,  the  Joint  Defense 
Manufacturing  Technology  Panel  (JDMTP)  established  a  working  group  to  develop  a  set  of 
Manufacturing  Readiness  Levels  that  were  synchronized  with  corresponding  TRL  levels  [7].  The 
initial  set  of  definitions  was  first  published  as  an  appendix  to  the  Technology  Readiness 
Assessment  Deskbook  in  2003. 
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Since  then,  a  comprehensive  website  has  been  developed  and  maintained  by  JTMDP  which 
contains  many  useful  tools,  guidelines,  and  references  associated  with  the  MRL  process  and 
associated  Manufacturing  Readiness  Assessments  (MRA),  including  reference  links  to  the 
Technology  Readiness  Assessment  Deskbook  [8].  The  gate  definitions  for  MRLs,  gate  exit 
criteria,  and  relationship  to  TRLs  and  the  Defense  Acquisition  System  phases  and  milestones,  are 
now  well  defined  and  are  succinctly  summarized  in  a  single-page  overview  chart  on  the  DoD 
MRL  website  [9], 

The  MRL  assessment  process  has  10  defined  readiness  levels,  as  shown  in  Figure  2,  above.  The 
following  comments  and  observations  are  made  in  the  context  of  the  present  Study  Task: 

•  MRL  1,  2  and  3  are  essentially  early  planning  gates,  and  are  tied  to  efforts  in  the 
corresponding  TRL  process  at  the  same  gates. 

•  MRL  4  is  a  key  gate  that  is  tied  to  Acquisition  Milestone  A  -  the  beginning  of  the 
Technology  Development  phase.  At  this  gate,  feasibility  has  been  demonstrated  in  a  lab 
environment,  and  key  processes  identified.  The  type  of  application  is  known,  but  specific 
system  requirements  may  not  yet  be  defined. 

•  MRL  5  and  6  gates  require  production  of  components  in  a  relevant  environment,  and 
upon  successful  completion  of  the  key  MRL-6  gate,  the  manufacturing  technology  is 
considered  mature  enough  to  be  included  in  the  EMD  phase  which  commences  at 
Milestone  B.  At  this  point,  specific  target  applications  are  well  defined  and  system 
requirements  are  known. 

•  MRL  7  and  8  gates  focus  on  process  maturity  and  preparation  for  production,  including 
process  demonstrations,  as  well  as  the  definition  and  validation  of  supply  chain  and 
quality  requirements.  The  key  gate  of  MRL-8  is  associated  with  Milestone  C,  the  start  of 
the  Production  and  Deployment  phase.  By  this  gate,  pilot-line  manufacturing  has  been 
successful,  and  the  processes  are  believed  ready  for  Low  Rate  Initial  Production  (LRIP). 

•  MRL  9  and  10  gates  represent  maturation  of  the  manufacturing  process(es)  through  low 
rate  and  finally  into  full-rate  production  capability,  including  process  monitoring  and 
statistical  process  control  (at  the  Six  Sigma  or  equivalent  level). 

•  The  MRL  gate  exit  criteria  are  closely  tied  to  the  corresponding  TRL  gates;  and  require 
that  the  corresponding  TRL  gate  has  been  passed. 

•  The  MRL  process  is  largely  reactive  to,  rather  than  interactive  with,  the  TRL  process. 

In  addition  to  the  well-defined  gates,  considerations,  and  exit  criteria,  nine  “Manufacturing 
Threads”  have  been  developed.  These  include  the  following: 

•  Technology  and  Industrial  Base 

•  Design 

•  Cost  and  Funding 

•  Materials 

•  Process  Capability  and  Control 

•  Quality  Management 

•  Manufacturing  Workforce 

•  Facilities 
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•  Manufacturing  Management. 

The  considerations  and  exit  criteria  for  MRL  gates  have  been  defined  for  each  of  the  nine 
manufacturing  threads  listed  above.  This  threaded  construct  provides  the  MRL  assessment 
process  with  an  organizing  structure  that  is  very  useful  for  both  industry  and  government. 
Questionnaires  have  been  developed  to  assist  Manufacturing  Readiness  Assessments,  to  identify 
and  focus  attention  on  activities  in  each  thread  that  are  necessary  to  achieve  a  desired  MRL  level, 
and  to  help  objectively  determine  current  MRL  for  subject  technologies. 

The  recent  cancellation  of  DODI  5000.02,  and  reissue  as  DODI  Interim  5000.02,  may 
significantly  affect  alignment  of  these  well-defined  MRL  gates  with  the  Defense  Acquisition 
System  key  milestones.  The  conceptual  program  management  models  represented  in  the  new 
“Interim”  instruction  will  very  likely  drive  actions  to  accelerate  MRL  maturation,  especially  for 
MRL  levels  of  5  and  beyond. 

Identification  of  Study  Focal  Areas  within  the  MRL  Process:  The  MRL  and  MRA  processes  are 
very  comprehensive  and  well  defined.  They  are  increasingly  used  by  industry  and  government, 
especially  at  the  system  level  for  major  programs  and  weapons  system  acquisitions.  In  such 
applications,  program  management  desires  to  assess  the  overall  technology  and  production 
readiness  for  the  entire  system;  thereby  focusing  on  the  manufacturing  readiness  assessment  of 
the  contractor’s  ability  to  produce  the  system  or  a  major  subsystem.  However,  there  may  be 
many  individual  technologies  involved,  each  with  different  challenges  and  risks.  This  is 
especially  a  concern  for  so-called  “critical  technologies”  -  those  that  are  included  in  the  system 
configuration  after  Milestone  B,  which  have  not  yet  achieved  TRL  and  MRL-6. 

Also,  there  is  certainly  a  desire  to  apply  the  MRL  process  to  assess  new  and  emerging 
technologies  -  perhaps  even  prior  to  a  specific  system  application  being  clearly  or  fully  defined. 
The  Study  Task  is  intended  to  identify  opportunities  and  methods  or  tools  that  could  generally 
accelerate  maturity  of  new  materials  and  manufacturing  process  technologies,  and  to  better 
identify,  reduce,  and  manage  risks  associated  with  such  technologies,  while  working  within  the 
current  MRL,  MRA,  and  TRL  processes. 

There  are  many  significant  efforts  currently  underway  in  both  government  and  industry  that  are 
relevant  to  this  study  task.  Examples  include  many  Integrated  Computational  Materials 
Engineering  (ICME)  projects  aimed  at  materials  and  process  modeling,  the  DARPA  Open 
Manufacturing  Initiative  -  with  several  performers  engaged  in  developing  methods  for  “rapid 
qualification”  of  materials  and  processes,  and  AFRL  sponsorship  of  efforts  to  develop  and 
exercise  Verification  and  Validation  methods  for  ICME. 

With  this  context  in  mind,  focus  areas  were  identified  for  further  assessment.  In  general,  these 
included  the  following  areas  of  consideration: 

•  Early  risk  identification  and  mitigation  approaches 

•  Increased  use  of  computational  modeling  for  materials  and  processes  (ICME),  including 
empirically  based  modeling 

•  Verification  and  Validation  methods  for  such  modeling 
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•  Uncertainty  Quantification  and  probabilistic  assessments 

•  Feedback  and  updating  methods  for  refining  models  based  on  early  data 

•  Increased  use  of  in-process  monitoring  and  advanced  inspection  methods  for  process 
monitoring  and  control 

•  Definition  and  execution  of  critical  experiments 

These  areas  were  generally  related  to  the  MRL  gates  and  to  the  nine  Manufacturing  Threads 
previously  cited,  to  ensure  compatibility  with  the  current,  accepted  process.  It  should  be  noted 
that  in  several  cases,  the  recommended  activities  list  associated  with  the  manufacturing  “threads” 
already  contain  some  of  the  items  identified  for  further  assessment  under  this  study  task.  For 
example,  “Production  modeling/simulation  approaches  identified”  is  contained  in  the  Production 
Capability  and  Control  thread,  beginning  with  MRL-4.  Also,  “In-process  and  final  inspections 
and  statistical  process  control  for  prototype  units”  is  contained  in  the  Quality  Management 
thread,  beginning  with  MRL-5.  Analyses  of  process  variation,  and  associated  process 
certifications  to  Six-Sigma  or  equivalent,  are  also  listed  activities,  but  they  occur  at  high  MRL 
levels.  In  general,  the  topics  identified  and  evaluated  under  the  study  task  are  recommended  as 
activities  which  begin  sooner  in  the  technology  maturation  process,  and  recommended  activities 
are  more  comprehensive  and  integrated  elements  than  are  considered  current  practice.  Also, 
references  for  established  practices,  guiding  documents,  or  procedures  are  recommended  to 
ensure  appropriate  application. 

3.4  Historical  Examples 

Several  prior  materials/processing  manufacturing  efforts  were  identified  and  reviewed  to  provide 
background  and  historical  reference  to  help  clarify  and  focus  ideas  within  this  study.  Given  the 
background  of  the  study  participants,  the  examples  focus  on  Aero-Propulsion  manufacturing 
technologies;  yet  these  examples  undoubtedly  share  characteristics  common  to  a  broader 
selection  of  technologies  important  to  DoD.  Table  1  outlines  some  of  the  key  issues  of  four 
manufacturing  development  efforts  that  encountered  significant  development  challenges.  The 
motivation  for  this  table  and  the  ensuing  analysis  is  to  identify  some  lessons  learned  to  inform 
the  identification  of  manufacturing  risks  earlier  and  with  greater  clarity. 

Lessons  Learned  about  Risks:  These  four  development  activities  were  undertaken  in  the  1970s 
and  1980s,  prior  to  the  emergence  of  formal,  structured  risk  assessment  processes  that  are  now 
commonly  exercised  by  both  industrial  and  governmental  organizations.  Additionally,  all  four  of 
the  historical  examples,  listed  in  Table  1,  represented  revolutionary  M&P  developments  that 
departed  significantly  from  the  former  mature  manufacturing  technologies  that  they  were 
intended  to  replace.  Because  these  new  manufacturing  technologies  broke  new  ground, 
developers  had  only  limited  knowledge  about  what  risks  to  expect  and  were  thereby  forced  to 
iteratively  react  to  problems  as  they  emerged,  with  subsequent  “fixes”  only  providing  limited 
success. 

Materials  engineers  have  long  understood  that  minimum  material  properties,  such  as  LCF  life, 
are  usually  govern  by  extreme  microstructural  features  and  particularly  material  defects. 

However,  a  large  number  of  tests  are  needed  to  identify  and  understand  extreme  defects 
governing  minimum  properties  (e.g.,  at  the  -  3  a  level);  and  as  such  significant  development 
resources  and  time  are  required  to  gauge  the  full  spectrum  of  defects,  their  individual  behaviors, 
and  design  potential  remediation  approaches. 
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Furthermore,  defects  revealed  during  initial  laboratory  development  (using  small  scale 
equipment,  clean  lab  environment,  and  highly  skilled  technicians)  often  are  not  the  same  as  those 
produced  within  a  production  environment.  Indeed,  scale-up  of  As-HIP  powder  processing  of 
rotor  hardware,  example  1  in  Table  1,  revealed  more  numerous  ceramic  defects  -  most  likely  due 
to  longer  atomization  runs  (melt-ceramic  crucible  interaction)  and  increased  contamination  from 
the  factory  environment  (e.g.,  dirty  air-lines,  inadequate  air  handling/filtration,  and  equipment 
degradation  owing  to  high  machine  duty-cycles  and  less  frequent/less  effective  machine 
cleaning/maintenance).  Indeed,  the  inadvertent  inclusion  of  such  contamination  resulted  in 
prior-particle  boundary  (PPB)  decoration  when  these  contaminants  chemically  reacted  during 
hot-isostatic -pressing  (HIP).  Furthermore,  the  size  of  these  PPB  defects  weren’t  constrained  by 
the  powder  sieve  size;  and  these  subtle  reactive  PPB  defects  had  low  probability  of  detection  via 
ultrasonic  inspection  because  their  volumetric  presence  was  considerably  lower  than  their  impact 
on  properties. 

Table  1:  Summary  of  prospective  benefits  and  risks  encountered  for  four  historical 


manufacturing  development  M&P  programs 


# 

M&P  Manufacturing 
Technology 

Key  Expected  Benefits 

Major  Risk  Issues 

1 

As-HIP  Powder 

Processing 

(superalloy  turbine  rotor 
components)  [10] 

-  Higher  y’  superalloys 

-  Reduced  manufacturing  cost 
owing  to  avoidance  of  billet 
working  and  forging 

-  Lower  cost  owing  to  expected 
reduction  in  the  number  of 
processing  steps 

-  Insufficient  powder  cleanliness 

-  Further  powder  contamination 
during  handling  -  including 
reactive  contaminants  that 
produced  Prior  Particle 

Boundary  (PPB)  defects  during 
HIP. 

-  NDT  inadequate  to  identify  key 
defects 

-  High  inspection  costs 

2 

Directionally  Solidified 
Superalloy  Eutectics 
(T urbine  blades)  [11,12] 

-  Higher  blade  operating 
temperature  compared  with 
directionally  solidified  blades 

-  Improved  longitudinal 
strength  and  creep  resistance 

-  3X  higher  cost  compared  to 

DSX  -  owing  largely  to  lower 
growth  rates  and  blades  per 
mold  cluster 

-  Lower  transverse  properties 

-  Poor  oxidation  and  HT 

corrosion  resistance 

3 

Spray  Forming 
(Superalloy  turbine  rotor 
components)  [13] 

-  Fine  grain  size  &  increased 
strength 

-  Avoidance  of  billet  working 
and  forging  operations 

-  Reduced  manufacturing  cost 

-  Low  (<  30%  process  yield)  due 

to  overspray  and  particle 
bounce 

-  Inadequate  process  control 

-  Porosity  from  gas  entrapment  & 

shrinkage 

4 

Continuous  Fiber  Ti 

Metal  Matrix  Composites 
(Aircraft panels  and  rotor 
components)  [14] 

-  Reduced  component  weight 

-  Higher  strength  along  fiber 
direction 

-  High  material  cost 

-  Low  transverse  LCF  and  FCG 

-  Lack  of  dimensional  control 

-  Fiber  diameter/stiffness  limited 
feasible  component  shapes 

All  four  historical  examples,  encountered  unexpected  cost  challenges.  For  example,  the  cost 
reduction  expected  for  As-HIP  superalloy  processing  was  lost  due  to  the  need  to  introduce 
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sophisticated  and  costly  quality  control  measures;  powder  inspection,  finish  part  contour 
following  ultrasonic  inspection,  and  the  adoption  of  low-cycle- fatigue  testing  as  a  specification 
requirement  all  added  both  cost  and  uncertainty.  In  contrast,  the  manufacture  of  directionally 
solidified  eutectic  blade  was  a  technical  success;  but  it  ultimately  succumbed  to  high  processing 
costs.  In  this  case  developers  understood  that  manufacture  of  eutectic  turbine  blades  required 
control  of  the  ratio  of  thermal  gradient  (G)  to  the  growth  rate  (R)  during  solidification.  To 
maintain  a  sufficiently  high  G/R  ratio  while  achieving  reasonable  productivity  meant  that 
developers  had  to  produce  a  high  gradient  within  the  furnace.  Unfortunately,  the  number  of 
blades  that  could  be  produced  simultaneously  in  the  directional  solidification  furnace  had  to  be 
significantly  reduced  and  yet  the  allowable  growth  rate  was  a  mere  0.25  inches  per  hour.  The 
resulting  blade  cost  was  greater  than  a  factor  of  three  compared  to  the  baseline  directionally 
solidified  dendritic  superalloy  turbine  blade  it  was  meant  to  replace. 

For  all  four  of  the  historical  development  examples,  the  overarching  risk,  as  alluded  to  at  the 
outset,  was  the  lack  of  up-front  knowledge  about  these  revolutionary  development  approaches. 
There  was  insufficient  insight  to  predict  or  even  imagine  potential  risk  sources  and  inadequate 
plans  at  program  inception  to  accelerate  acquisition  of  required  knowledge.  It  is  tempting  to 
wonder  how  modem  risk  assessment  methods  might  have  changed  the  trajectories  and  final 
results  of  these  historical  manufacturing  developments.  However,  even  today  it  can  be 
extremely  challenging  to  quickly  identify,  analyze,  and  resolve  previously  unknown  and 
unexpected  risks,  particularly  in  new  manufacturing  domains. 

3.5  Early  Risk  Identification  and  Mitigation 

Plans  for  new  Manufacturing  Technology  (MT)  development,  even  those  that  pursue  seemingly 
excellent  opportunities  too  often  have  encountered  major  risks  that  disrupt  the  development 
process  trajectory,  lessen  the  benefit  of  the  new  MT,  and/or  cause  program  schedule  and  cost 
problems.  As  expressed  within  the  MRL  Deskbook: 

“The  purpose  of  an  MRL-based  assessment  of  manufacturing  readiness  is  to  analyze 
current  conditions  and  to  identify  manufacturing  risk  in  order  to  assist  the 
program/project  manager  in  creating  a  plan  or  options  to  reduce  or  remove  risks.  ” 

Section  3.4  of  this  report  presented  four  historical  example  MT  development  programs,  each  of 
which  encountered  unexpected  risks  that  then  led  to  either  the  suspension  of  further  development 
and/or  terminated  implementation.  Of  course,  most  MT  development  programs  don’t  encounter 
such  extreme  results.  But  most  MT  development  programs  do  encounter  risks  that  while  not 
necessarily  debilitating  can  disrupt  MT  development  progress  by  requiring  expanded  and 
extended  research,  modifying  or  adding  a  process  steps,  or  causing  retrenchment  from  original 
program  goals,  payoff,  and  some  implementation  opportunities.  These  historical  examples, 
though  seemingly  extreme,  do  provide  a  useful,  cautionary  perspective,  mindset,  and  “meta 
lessons”  that  MT  developers  should  consider  to  avoid  or  lessen  similar  risks  in  their  programs. 

Figure  5  depicts  how  risks  can  infect  a  manufacturing  technology  development  program  either 
through  (1)  inadequate  MT  process  strategy  &  design,  and/or  (2)  deficient  process  and  quality 
controls  -  as  shown  at  the  bottom  and  top  of  the  figure,  respectively.  The  two  sets  of  boxes 
(outlined  in  red)  highlight  common  risk  sources,  based  on  historical  examples,  though  it  is  very 
unlikely  that  all  will  be  encountered  in  a  typical  manufacturing  technology  development 
program.  Obviously  identification,  assessment,  and  mitigation  of  risk  should  focus  on  those 
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process  steps  that  are  new  and  thus  less  well  known  to  the  development  team.  Of  course,  the 
phrase  “less  well  known  to  the  development  team  ”  begs  several  questions  that  will  be  addressed 
below. 


MT  development  program  success  is  enhanced  when  developers  proactively  focus  research  and 
testing  efforts  to  uncover  technical  and  cost  risks  and  then  address  and  resolve  them  quickly. 

But  unfortunately,  some  MT  development  teams  instead  unwittingly  delay  risk  identification  and 
instead  focus,  single  mindedly,  on  activities  intended  to  “prove”  positive  MT  research  claims  and 
consequent  projected  benefits.  As  the  historical  examples  have  shown,  failure  to  identify  and 
then  quickly  resolve  MT  development  risk  can  increase  the  risk  of  program  failure. 
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Figure  5:  Diagram  showing  an  idealized  manufacturing  process  under  development  and 
the  associated  development  and  control  deficiencies  that  can  increase  risk 


Unfortunately,  risk  perception  acuity  for  novel  manufacturing  technologies  depends  on  the 
breadth  of  experience  and  the  outlook  (advocacy  versus  skepticism)  of  members  of  the 
Integrated  Product  Development  Team  (IPDT)  -  as  well  as  the  acceptance  of  contrarian 
viewpoints  by  the  team  and  its  leaders.  Consider  the  three  categories  of  MT  process 
developments  risk,  listed  below1: 

1 .  Known  MT  process  risks  that  are  understood, 

2.  Known  MT  process  risks  that  are  not  understood,  and 

3.  Unknown  MT  process  risks  that  are  not  understood,  even  once  discovered. 

The  first  risk  category  addresses  risks  that  are  benign  when  addressed  actively.  Risks  of  this  type 
should  emerge  during  a  formal  risk  assessment  conducted  by  the  team;  and  risk  confirmation  and 
subsequent  mitigation  should  be  straightforward  because  they  are  understood.  Risks  associated 
with  the  second  category,  like  the  first,  should  be  identified  as  possible  risks  during  a  formal 
IPDT  risk  assessment.  However,  risk  candidates  in  this  second  category  will  be  more  difficult 
to  confidently  confirm,  quantify,  or  mitigate  because  understanding  is  lacking.  Instead,  if  the 


1  A  variant  of  former  U.S.  Secretary  of  Defense  Donald  Rumsfeld’s  well  known  quote  regarding  “  Unknown  Unknowns,”  or 
“Unk-Unks” 
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team  judges  these  risks  to  possibly  have  sufficient  likelihood  of  occurrence  and  impact,  the  IPDT 
will  need  to  undertake  targeted  research  to  fill  the  associated  knowledge  gap.  The  third  risk 
category  poses  the  greatest  challenge  to  a  MT  development  team,  particularly  regarding  lurking 
risks  that  have  low  occurrence  rates  and  moderate  to  high  impact.  These  lurking  risks, 
sometimes  referred  to  as  “black  swans”,  cannot  be  identified  using  passive  risk  assessment  and 
instead  will  emerge  only  during  MT  development  experiments  or  trials  for  teams  that  fail  to 
exercise  active  risk  identification.  If  discovered  during  later  stages  of  MT  development,  such 
risk  events  can  have  a  debilitating  effect  on  program  success,  cost,  and/or  schedule.  Also, 
because  this  category  of  risk  is  also  not  understood,  like  the  second  category,  research  will  be 
required  to  fully  confirm,  quantify,  and  mitigate  these  risks  once  they  become  known.  Of 
course,  if  the  risk  does  not  appear  until  after  prototype  MT  development  has  been  completed, 
development  will  be  disrupted  further  because  research  experiments  will  likely  be  more  difficult 
to  execute  in  a  near-production  environment. 

There  is  one  further  point  regarding  the  second  and  third  categories  of  risk,  alluded  to  earlier. 

The  phrases  “unknown  MT  process  risks”  and  “are  not  understood”  each  beg  the  question  of 
“who  doesn’t  know”  and  “who  doesn’t  understand”.  Of  course  the  “who”  is  the  MT 
development  team;  but  other  experts,  outside  the  IPDT,  may  have  direct  knowledge  or  access  to 
other  resources,  possibly  from  similar  or  related  processes  outside  the  purview  of  the  MT 
development  team.  Also,  independently,  the  IPDT  can  attempt  to  expand  their  knowledge  of 
unfamiliar  process  step(s)  by  identifying  analogous  process  step  via  patent,  literature  research,  or 
ICME  modeling.  Consequently,  identifying  category  2  and  3  risks  can  be  aided  by  the  following 
strategies: 

•  Involving  independent  experts  to  augment  the  IPDT  depth  and  breadth  of  knowledge; 

•  Accelerating  MT  experimental  research,  adapting  and  applying  Integrated  Computational 
Materials  Science  and  Engineering  (ICMSE),  and  implementing  uncertainty 
quantification  (UQ)  for  lesser  known  MT  sub-processes;  and 

•  Adding  focus  on  increasing  the  robustness  for  sub-process  that  exhibit  high  variability 
during  early  MT  experiments  and  trials. 

The  first  strategy  in  the  above  list,  involving  outside  experts,  may  appear  to  some  as  being 
counter-intuitive  -  after  all,  wouldn’t  an  IPDT  pursuing  a  state-of-the-art  MT  development 
include  the  necessary  and  pertinent  expertise  to  successfully  execute  the  MT  development 
program?  In  many  cases  this  may  be  true,  but  not  necessarily  for  all  MT  development  programs. 
Most  technologies  (including  MT)  advance  via  evolution;  and  often  these  advances  are  inspired 
and  propelled  by  technology  transfer  from  other  processes  and  industries  -  directly,  indirectly,  or 
diffusely,  via: 

•  Refining  existing  processes  (via  additional  experimentation,  modeling,  control,  process 
design  refinements); 

•  Incorporating  new  sub-processes  transferred  from  or  inspired  by  external  MT  innovation 
in  other,  related  industries;  and/or 

•  Adopting  a  “new  generic  approach”  developed  in  another  field  to  advance  the  current 
manufacturing  technology. 
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If  any  of  these  three  scenarios  are  applicable,  the  implication  is  that  even  when  an  MT 
development  team  is  unexpectedly  about  to  confront  a  “Known-Unk”  or  “Unk-Unk”  risk: 

•  There  may  be  outside  experts  that  have  already  confronted  these  or  similar  risks; 

•  These  outside  experts  may  have  knowledge  about  physical  mechanisms,  relevant  models, 
NDE  methods,  QC  testing  procedures,  uncertainty  quantifications,  or  effective 
remediation  approaches;  and 

•  Independent  outside  experts  (involved  with  predecessor  technologies)  can  provide 
unbiased  assessment  of  risk  magnitude,  and  guidance  about  risk  mitigation  strategies  and 
techniques  as  well  recommendation  regarding  MT  technical  program  modification. 

The  key  bottom  line  is  that  an  unknown  risk  lurking  within  a  MT  development  program,  in  some 
instances,  may  be  evident  and  understood  with  input  from  researchers  and  engineers  outside  the 
program;  often  gained  through  expert  elicitation  processes. 

3.6  Focus  Areas  Identified  for  Evaluation 

Based  on  the  cumulative  finding  of  the  present  study,  we  believe  that  the  advance  of 
manufacturing  technology  can  be  accelerated  with  greater  confidence  by  implementing  an 
integrated  set  of  methods  and  practices,  which  were  identified  and  evaluated  under  this  Study 
Task.  These  included  the  following  areas  of  consideration: 

•  Early  risk  identification  and  mitigation  approaches 

•  Increased  use  of  computational  modeling  for  materials  and  processes  (ICME),  including 
empirically  based  modeling 

•  Verification  and  Validation  methods  for  such  modeling 

•  Uncertainty  Quantification  and  probabilistic  assessments 

•  Feedback  and  updating  methods  for  refining  models  based  on  early  data 

•  Increased  use  of  in-process  monitoring  and  advanced  inspection  methods  for  process 
monitoring  and  control 

•  Definition  and  execution  of  critical  experiments 

These  topics  were  identified  in  this  study  as  key  factors  to  both  accelerate  the  acquisition  of 
essential  knowledge,  provide  an  earlier  and  more  accurate  identification  of  potential  MT 
development  risks,  and  engage  a  suite  of  activities  intended  to  reduce  risk  and  accelerate 
technology  maturation.  Each  of  these  tactics/tools  is  described  greater  detail  in  the  following 
sections. 

3.6.1  Expert  Elicitation.  Expert  elicitation  (EE)  is  typically  carried  out  using  standard  team 
processes,  such  as  the  Phenomena  Identification  and  Ranking  Technique  (PIRT)  and  the  Failure 
Modes,  Effects,  and  Criticality  Analysis  (FMECA)2’ 3  by  an  independent  team  composed  of 
experts  from  within  and/or  outside  the  companies  and  organizations  engaged  in  the  MT 
development  effort. 


2 

The  PIRT  and  FMECA  processes  will  be  described  in  greater  detail  in  the  Appendix  to  this  report. 

There  are  several  other  potentially  useful  methods  including  the  Delphi  Process. 
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Based  on  the  selected  EE  process  and  targeted  MT  process  step(s),  EE  team  membership  should 
be  selected  such  that  their  expertise  spans  the  relevant  disciplines  and  industrial  processes 
associated  with  the  MT  development  and  output  product.  The  span  of  expertise  might  include 
individuals  with  in-depth  experience  and  knowledge  in:  manufacturing,  materials  &  processing 
technology,  ICMSE  modeling,  quality  control,  and  cost  analysis,  among  others  that  are  germane 
to  the  specific  MT  under  development. 

The  two  EE  processes  selected  for  focus,  PIRT  and  FMECA,  employ  much  different,  yet 
complementary,  lines  of  reasoning.  Both  EE  techniques  focus  on  MT  process  steps  that  are 
important,  but  less  well  known  relative  to  process  behavior  and  product  characteristics.  These 
two  EE  techniques  are  introduced  in  brief  below  with  greater  detail  available  in  the  appendix  and 
via  report  references: 

•  PIRT  is  executed  by  first  listing  phenomena  operative  within  targeted  process  steps.  The 
EE  team  then  judges  and  scores  how  important  each  phenomenon  might  be  in  influencing 
manufacturing  system  performance  and  product  requirements,  as  well  as  the  level  of 
knowledge  for  each  the  phenomenon.  Subsequently,  “importance”  and  “knowledge” 
scores  are  combined  and  assessed  to  identify  both  the  critical  phenomena  and  MT  research 
that  should  be  considered  to  fill  key  knowledge  gaps. 

•  FMECA  starts  by  listing  potential  failure  modes  for  each  targeted  process  step.  Much  like 
for  the  PIRT  process,  the  EE  team  then  (1)  lists  potential  effects  caused  by  the  failure  and 
(2)  judges  and  scores  each  potential  failure  mode  relative  to  probability  of  failure,  severity 
of  the  effect,  and  probability  of  detecting  the  failure  or  effect.  After  all  potential  failure 
modes  are  assessed  and  prioritized,  the  EE  team  recommends  remedial  actions. 

Implementation  of  EE  can  have  far  reaching  impact  on  MT  development  program  success, 
particularly  if  it  is  initiated  in  the  early  stages  of  development.  Much  of  the  preceding 
discussion  has  focused  or  risk  identification  and  assessment;  however,  EE  benefits  extend, 
either  directly  or  indirectly,  to  the  other  systems  engineering  strategies  and  tools 
recommended  by  this  study.  All  such  down-stream  benefits  accrue  through  the 
enlightening  effect  of  knowledge  gained  early  enough  to  inform  downstream  decisions, 
which  thereby  should  be  improved,  yielding  greater  process  understanding.  In  a  real  sense, 
EE  provide  the  following  benefits  beyond  identifying  key  MT  development  risks,  as  shown 
in  Figure  6: 

Near  Term  (MRL  1-4): 

•  Identify  manufacturing  technology  process  gaps  and  help  focus  ensuing  MT  research; 

•  Guide  ICMSE  support  teams  to  emphasize  development  of  models  that  address  high 
priority  process  steps  and  associated  materials  models;  in  turn,  system  engineering 
techniques  can  use  these  models  to  determine  process  parameter  sensitivities,  conduct 
analytical  DOE  analyses,  and  support  uncertainty  quantification  calculations; 

•  Recommend  critical  material  property  measurements  and  experiments  to  better  define 
process  boundary  conditions,  process  development  trials,  and  important  process  parameters 
that  should  be  measured  and  analyzed  -  particularly  to  support  QC  development  efforts 
and  aid  uncertainty  quantification  assessments;  and 
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•  Offer  guidance  on  prospective  NDE  techniques  and  associated  detection  POD  issues  that 
may  lie  beyond  the  experience  of  the  MT  development  team; 

Later  Term  (MRL  4  -  7): 

•  Review  and  critique  MT  development  team  efforts  to  verify  and  validate  (V&V)  MT 
program  ICMSE  and  uncertainty  quantification  (UQ)  thrusts; 

•  Support  manufacturing  technology  maturation  and  provide  input  to  MT  development 
planning  and  scale-up; 

•  Provide  input  to  identify  production  QC  and  NDE  needs  and  techniques;  and; 

•  Provide  guidance  to  help  define  and  assess  process  control  targets,  strategies,  and  relevant 
sensor  and  control  technologies. 

Achieving  these  additional  benefits  will  require  that  the  MT  IPDT  determine  where  the  EE 
team  can  provide  benefit  and  then  convene  the  EE  team  at  critical  MRL  junctures. 
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•  Aiding  definition  of  ICMSE  V&V  and  UQ 
approach 

•  Supporting  manufacturing  technology 
maturation  and  scale-up  planning 

•  Helping  to  identify  QC  and  NDE  needs 
and  methods 

•  Helping  to  define  process  control  targets 
and  relevant  sensor  &  control 
technologies 


Figure  6:  Outline  of  benefits  that  expert  elicitation  provides  to  an  MT  development  effort 

3.6.2  Computational  Models  for  Materials,  Manufacturing  Technology,  and  Processes. 

Three  areas  identified  for  further  evaluation  are  closely  related.  These  are: 

•  Increased  use  of  computational  modeling  for  materials  and  processes  (ICME),  including 
empirically  based  modeling 

•  Verification  and  Validation  methods  for  such  modeling 

•  Uncertainty  Quantification  and  probabilistic  assessments 

Increased  use  of  computational  modeling  for  materials  and  processes  (ICME):  The  potential  for 
ICME  to  provide  significant  benefit  to  materials  and  processing  technology  development  has 
been  cited  by  numerous  studies  at  the  national  level.  These  include  a  landmark  2008  National 
Research  Council  study  identifying  ICME  as  a  transformational  discipline  and  essential  future 
capability  [15],  a  follow-on  National  Research  Council  study  regarding  importance  of  ICME  for 
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lightweighting  technology  application  [16],  and  a  TMS  sponsored  study  regarding  how  to 
achieve  pervasive  ICME  implementation  [17].  It  is  also  a  primary  focus  and  enabler  for  the 
White  House’s  Office  of  Science  and  Technology  Policy  (OSTP)  Materials  Genome  Initiative 
[18].  The  means  to  align  ICME  with  standard,  gated  development  processes  including  TRL, 
MRL,  IPD,  and  the  DoD  Defense  Acquisition  Guide  milestones  and  process,  have  been 
described,  and  related  to  reduced  time,  cost  and  risk  of  new  materials  and  manufacturing 
technology  implementation. 

ICME  is  now  an  integral  part  of  many  government  agency  and  industrial  development  programs. 
There  are  now  available  a  number  of  commercial  codes  and  sources  for  various  aspects  of  an 
ICME  system.  These  include  codes  for  thermodynamic-based  materials  analysis  and  predictions, 
various  process  models  for  casting,  forging,  and  welding,  and  software  capability  to  integrate 
various  independent  models  with  design,  thermal,  and  structural  analysis  tools.  Some  successful 
applications  of  comprehensive  ICME  have  been  documented  in  the  literature,  but  to  date  the 
broad  implementation  and  integration  of  ICME  with  design,  analysis,  manufacturing,  and  quality 
disciplines  has  not  been  realized  -  at  least  not  in  general,  standard  practice.  There  are  several 
initiatives  underway  that  are  either  focused  on  accelerating  ICME  implementation,  or  are 
depending  on  ICME  benefits  to  accelerate  materials  or  manufacturing  technology  maturation. 
Two  with  special  relevance  to  this  study  task  are: 

•  AFRL  sponsored  Foundational  Engineering  Problem  (FEP)  initiatives  for  implementation 
of  selected  ICME  applications.  Of  particular  interest  is  the  Metals  Affordability  Initiative 
(MAI)  program,  P&W-8,  for  Incorporation  of  Residual  Stresses  in  the  Design  of  Nickel- 
Based  Superalloy  Structures. 

•  The  DARPA  sponsored  Open  Manufacturing  initiative,  specifically  those  programs 
focused  on  rapid  qualification  of  additive  manufacturing  processes.  DARPA  Open 
Manufacturing  is  the  first  known  program  where  ICME  and  related  support  activities, 
listed  above,  have  been  systematically  incorporated  as  a  means  to  accelerate 
manufacturing  process  qualification,  and  reduce  risk  of  implementation  of  new  materials 
and  manufacturing  technology. 

In  the  ideal  state,  an  ICME  system  will  encompass  comprehensive  quantitative  modeling  of  a 
material  -  whether  metal  alloy  or  composite  -  including  its  composition,  processing,  resulting 
microstructure,  and  finally  the  mechanical  behavior  and  properties  of  the  material  relevant  to  the 
application  of  interest.  While  there  are  many  challenges  in  the  development  of  the  integrated  and 
linked  models  that  comprise  an  ICME  system,  the  capability  has  been  demonstrated,  and  early 
results  from  the  DARPA  Open  Manufacturing  programs  have  been  successful.  The 
computational  models  that  comprise  an  ICME  system,  and  the  supporting  data  used  to  generate 
the  models,  will  facilitate  much  more  direct  integration  of  materials  and  manufacturing  processes 
with  the  design,  structural  analysis,  life  prediction,  and  quality  disciplines.  Development  and 
implementation  of  comprehensive,  validated  ICME  methods  and  models  will  reduce  dependence 
on  lengthy,  expensive  test  programs,  reduce  the  number  of  development  iterations,  accelerate 
process  optimization,  enable  tailoring  to  design  and  structural  requirements,  and  guide  informed 
process  control  and  accelerated  process  qualification. 
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Verification  and  Validation  methods:  For  an  ICME  application  to  be  successful  and  credible  for 
decision  making,  rigorous  verification  and  validation  processes  are  essential.  This  is  truly  a 
critical  requirement  to  ensure  that  program  management,  especially  the  chief  engineers  for  the 
end  product  or  application,  and  regulatory  authorities,  have  confidence  in  the  fidelity,  maturity, 
and  validation  of  model-based  decisions. 

The  materials  and  processing  discipline  has  been  regarded  less  mature  than  other,  more 
computationally  focused  engineering  disciplines,  in  the  area  of  rigorous  V&V  for  computational 
models.  Rigorous  verification  and  validation  methods  have  been  developed  for  other  engineering 
disciplines  -  notably  the  2006  ASME  Guide  for  Verification  and  Validation  for  Computational 
Solid  Mechanics  [19].  This  guide  was  the  product  of  many  years  of  continuous  industry  and 
agency  efforts,  and  had  the  benefit  of  ASME  committee  focus  to  ensure  a  comprehensive  and 
rigorous  process.  AFRL  recognized  the  importance  of  V&V  to  future  success  of  ICME,  and  has 
supported  several  efforts  aimed  at  developing  guidelines  and  a  recommended  best  practice 
specifically  for  ICME  V&V  and  aerospace  applications.  Guidelines  and  a  recommended  best 
practice  were  developed  which  extensively  referenced  and  incorporated  the  ASME  V&V  Guide 
10-2006.  The  ASME  V&V  Guide  was  supplemented  for  specific  ICME  V&V  application: 
practitioner  aides  were  developed  in  the  form  of  checklists  for  V&V  planning  and  progress 
assessment,  a  Tool  Maturity  Level  (TML)  approach  for  assessing  maturity  of  ICME  methods  and 
models,  and  a  recommended  approach  for  risk  vs.  consequence  assessment  related  to  use  of 
ICME  models  for  specific  applications.  [20] 

It  is  important  to  note  that  the  guidelines  and  the  practitioner  aides,  including  checklists  and  Tool 
Maturity  Level  assessment  guide,  are  all  aligned  and  integrated  with  established  technology  and 
system  development  gated  processes  -  including  TRL,  MRL,  IPD,  and  the  Defense  Acquisition 
Guidebook.  This  is  important,  because  ICME  and  process  model  V&V  efforts  must  support  the 
level  of  risk  and  consequence  associated  with  specific  applications.  Alignment  with  the 
controlling,  gated  practices  for  technology  and  product  development  is  critical  to  acceptance  by 
program  management,  chief  engineers,  and  responsible  DoD  or  regulatory  agencies.  Recent 
updates  to  the  guidelines  and  recommended  best  practice  have  been  generated,  and  this  version 
was  approved  by  AFRL  for  public  release,  Distribution  A,  in  January,  2014  [21].  The  excel  files 
with  practitioner  aides  and  recommended  best  practice  summary  has  also  been  revised  and 
approved  for  public  release,  Distribution  A  [22],  (Note:  the  site  where  these  documents  will 
reside  for  future  reference  purposes  has  not  yet  been  determined  -  but  they  are  available  upon 
request  from  the  authors). 

Uncertainty  Quantification:  One  critical  element  of  ICME  model  development,  V&V,  and 
process  qualification,  is  Uncertainty  Quantification  (UQ).  UQ  represents  assessment  of  model 
input  parameters  and  their  known  or  expected  variation,  followed  by  quantitative  exercise  of  the 
computational  model  (or  models)  in  a  manner  that  permits  assessment  of  the  sensitivity  and 
response  of  the  model  to  the  input  parameters  over  the  range  of  intended  application.  The  result 
is  a  quantitative  assessment  of  the  uncertainty  of  the  model  and  its  sensitivity  to  parameters  of 
interest.  This  is  a  complex  but  essential  activity  for  guiding  experimental  efforts  to  support 
model  development  and  validation,  as  well  as  to  guide  model  refinements  that  may  be  needed  to 
achieve  desired  or  acceptable  fidelity.  The  approach  and  key  steps  for  UQ  are  described  in  the 
ASME  V&V  Guide  10-2006  [19].  Recommendations  for  use  of  UQ  and  other  probabilistic 
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assessments  are  included  in  the  Guidelines  and  Recommended  Best  Practice  for  ICME  V&V 
previously  cited.  These  are  dependent  upon  the  assessment  of  Tool  Maturity  Level  (TML),  and 
of  course  are  related  directly  to  the  intended  ICME  application 

Both  the  AFRL  FEP  on  Nickel  Residual  Stress,  and  the  Open  Manufacturing  programs  have 
clearly  demonstrated  that  the  necessary  probabilistic  tools  and  methods  to  support  UQ  are 
available  and  capable.  In  addition,  the  requisite  expert  practitioner  skills  are  either  available 
directly  in  the  contract  performer  organizations,  or  are  available  in  outside  organizations  such  as 
Southwest  Research  Institute  (SwRI)  and  universities.  The  results  of  UQ  assessments  are 
essential  to  guide  model  development,  refinement,  and  validation  efforts  towards  achieving 
acceleration  of  materials  and  manufacturing  technology  maturation,  and  reducing  risk  of 
implementation. 

Recommendations  Resulting  From  Study  Task:  Given  that  increased  development  and  use  of 
ICME  for  materials  and  manufacturing  process  models  offers  significant  potential  benefit  for 
accelerated  maturation  of  manufacturing  technologies,  when  supported  by  the  requisite  V&V  and 
UQ  supporting  activities,  the  key  questions  are: 

1 .  How  much  computational  (or  hybrid  computational  and  empirical)  modeling  is 
appropriate,  or  optimal,  and 

2.  How  to  align  ICME  model  development  and  validation  with  the  MRL  and  MRA 
processes? 

The  current  Process  Capability  and  Control  thread  lists  “initial  models  developed,  if  applicable,” 
as  early  as  MRL-1,  a  high  level  manufacturing  process  flow  chart  is  called  for  at  MRL-2, 
“production  modeling/simulation  approaches  identified”  is  included  as  an  MRL  4  gate 
completion  requirement.  Additional  descriptions  of  model/simulation  progress  follow  under 
subsequent  MRL  gates.  It  appears  that  manufacturing  variation  is  determined  relatively  late 
(MRL-7+),  mostly  by  experimental  measurement  or  demonstration  of  process  output.  It  appears 
that  the  modeling  and  simulation  efforts  are  largely  targeted  for  eventual  support  of  low  rate  or 
full  production. 

Recommendations  from  this  study  task  are  consistent  with  these  current  listings;  but  propose 
broader  modeling  and  simulation  expectations,  close  coupling  with  V&V  and  UQ  activities,  and 
more  integrated  use  of  these  models  with  design,  materials,  structures  (analysis)  and  quality 
requirements.  Equally  important,  it  is  believed  that  early  development  and  continuous 
refinement  of  ICME  and  manufacturing  process  models,  including  assessment  of  variation,  must 
be  an  integral  part  of  accelerated  maturation  efforts.  While  these  items  may  already  appear  in 
the  various  Manufacturing  Threads,  the  Study  Task  recommends  earlier  application,  and 
provides  some  added  definition  and  detail. 

The  previously  described  alignment  of  ICME  with  current  gated  processes,  including  TRL, 
MRL,  IPD,  and  Defense  Acquisition  Guidebook  Milestone  requirements  (actually  detailed  in 
Interim  DoDI  5000.02,  issued  November  2013)  is  regarded  appropriate  at  this  time.  The  ICME 
V&V  Guidelines  and  Recommended  Best  Practice  [21,  22]  is  also  believed  appropriate, 
although  the  practitioner  aides  for  Model  and  System  Level  Checklists,  and  the  Tool  Maturity 
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Level  (TML)  assessment  guide  may  need  revision  for  specific  application  to  manufacturing 
technology  and  consistency  with  the  MRL  process. 

One  important  recommendation  is  that  an  ICME  and  manufacturing  process  modeling  strategy 
be  developed  beginning  very  early  -  in  the  MRL  1  to  3  period.  This  may  be  initiated  as  a  simple 
flow  diagram,  detailing  expected  processes  and  steps  for  a  manufacturing  technology  under 
development.  The  simple  flow  diagram  can  be  evolved  as  development  proceeds,  incorporating 
more  detail,  identifying  where  computational  modeling  is  appropriate,  and  identifying  key 
model  inputs,  required  supporting  data,  key  model  outputs,  and  requirements  for  integration 
with  other  engineering  disciplines  and  models.  Utilization  of  the  practitioner  aides  in  the  ICME 
V&V  Guidelines  and  Recommended  Best  Practice  [22]  provide  assistance  in  terms  of  model 
and  system  level  checklists,  Tool  Maturity  Level  assessment  guidance,  and  use  of  tools  such  as 
PIRT  for  assessing  key  model  parameters  and  hierarchy  of  sub-models.  The  initial  assessment  of 
UQ  and  development  of  a  V&V  plan  may  occur  later  -  in  the  MRL  4  to  6  period  perhaps,  and 
execution  of  the  V&V  plan  and  completion  may  occur  even  later. 

The  key  recommendation  is  that  a  computational  modeling  plan  and  strategy  be  developed  early, 
with  expectation  that  it  will  be  refined  and  detailed  as  technology  development  proceeds. 
Recommendations  for  when  and  how  to  do  this  within  the  current  process  are  described  under 
Section  4  of  this  report. 

3.6.3  Feedback  and  Updating  Processes.  Experimental  data  is  continuously  generated  as  a 
normal  part  of  Manufacturing  Technology  and  supporting  ICME  developments.  This 
experimental  data  is  important  for  demonstration  and  quantification  of  new  process  capability 
and  output,  from  basic  material  microstructure  and  properties  to  complex  measures  of  capability 
and  structural  integrity,  including  relevant  defect  species.  It  is  also  essential  to  define  process 
limits  and  controls,  to  validate  sensitivity  and  effect  of  key  process  parameters,  and  ultimately  to 
validate  and  support  process  qualification  for  intended  applications.  This  data  is  also  important 
for  another  purpose:  a  rigorous  and  systematic  updating  process  to  refine  and  improve  ICME 
models  and  parameters,  including  their  variation,  which  are  deemed  important  to  represent  the 
materials  and  manufacturing  processes  of  interest.  Ideally,  this  would  be  a  continuous  or  periodic 
closed  loop  process,  and  would  continue  beyond  process  qualification  into  the  production  phase 
of  system  acquisition.  The  updating  process,  whether  based  on  the  Bayesian  inference  method 
and  function,  or  an  alternative  method,  is  essential  to  achieve  a  rapid  maturation  and  validation 
of  both  the  ICME  models  which  may  be  employed,  as  well  as  key  process  parameters  and 
controls  that  may  be  identified  and  used.  This  may  differ  substantially  from  legacy  approaches, 
which  often  involve  defining  a  process  with  empirical  substantiation,  establishing  a  “frozen 
process,”  followed  sequentially  by  extensive  experimental  characterization  of  process  output.  In 
the  not-uncommon  event  that  some  iteration  in  material,  processing,  or  the  end  application 
becomes  necessary  -  significant  repetition  of  the  experimental  characterization  program  may  be 
required.  The  intent,  and  a  major  benefit,  of  ICME-enabled  technology  development,  with 
appropriate  V&V,  UQ,  and  systematic  updating  of  models  and  data  bases,  is  to  minimize  both 
the  occurrence  and  impact  of  such  development  iterations. 

The  current  Manufacturing  Threads  include  mention  of  designed  experiments  (DFX)  relatively 
late  in  the  process  -  at  MRL-6  and  7  under  the  Design  thread,  although  use  of  statistical  process 
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control  appears  earlier  at  MRL-5  under  the  Quality  Management  thread.  What  is  recommended 
as  a  result  of  this  study  task  is  systematic  and  disciplined  use  of  updating  and  refinement 
methods  for  both  the  computational  modeling  efforts  and  the  manufacturing  processes 
themselves,  beginning  in  the  MRL  -4  to  5  time  period  and  continuing  thereafter.  It  is  believed 
that  such  practices  can  accelerate  maturation  of  new  technologies,  and  accelerate  achievement  of 
statistical  process  control  and  realization  of  “six-sigma”  process  capability,  before  production 
commences  at  MRL-9  to  10. 

3.6.4  In-Process  Monitoring  and  Advanced  Inspection  Techniques.  The  advanced  state  of 
information  technology  for  data  acquisition  and  analysis,  coupled  with  today’s  advanced  sensor 
and  inspection  technologies,  form  the  basis  for  another  element  for  accelerating  maturation  and 
reducing  risk  of  new  materials  and  manufacturing  process  technology.  In-process  quality 
monitoring,  and  advanced  inspection  capability,  offer  independent  methods  to  validate  process 
output,  ensure  processes  are  operating  within  defined  limits,  and  even  provide  potential  for 
closed-loop  control  of  key  process  parameters  that  was  not  previously  possible.  While  such 
capability  would  be  very  beneficial  as  independent  elements,  they  can  be  especially  effective 
when  coupled  with  ICME-enabled  material  and  process  modeling,  and  uncertainty  quantification 
of  key  model  parameters  and  outputs.  The  DARPA  Open  Manufacturing  programs  addressing 
Rapid  Qualification  have  invested  in  advanced  inspection  methods  and/or  integrated  in-process 
monitoring,  as  a  means  of  achieving  rapid  qualification  goals.  These  methods  address  a 
significant  goal  of  the  Open  Manufacturing  program  for  rapid  qualification  success:  to  develop 
significant  understanding  of  process  capability,  performance,  and  limits  that  qualification  for 
various  applications  can  be  greatly  accelerated,  and  implementation  risks  greatly  reduced. 

The  current  manufacturing  thread  descriptions  list  identification  of  “in-process  and  final 
inspections”  at  MRL  4  and  5;  and  these  are  later  associated  with  Key  Process  Characteristics 
(KPC’s)  at  MRL-6,  under  the  Quality  Management  thread.  The  use  of  Statistical  Process  Control 
is  also  contained  in  the  Quality  Management  thread,  beginning  with  prototype  manufacturing  at 
MRL-5.  This  Study  Task  advocates  increased  focus  on  in-process  quality  assurance  that  involves 
identification,  direct  measurement,  and  monitoring  of  key  process  control  parameters.  It  is 
important  to  then  integrate  this  accumulated  information  with  ICME  development  to  improve 
models  and  gain  better  assessment  of  uncertainty  associated  with  modeling  errors  and  process 
variation. 

The  Study  Task  recommends  MT  developers  use  in-process  monitoring  and  advanced  inspection 
techniques,  beginning  early  in  the  MRL  process.  This  approach  can  accelerate  maturation  of  new 
technologies,  achievement  of  (1)  statistical  process  control,  (2)  accelerated  realization  of  “six- 
sigma”  process  capability,  and  (3)  add  substantial  confidence  in  new  materials  and 
manufacturing  technologies.  These  results  can  be  achieved  largely  by  providing  appropriate 
objective  measures  for  key  characteristics  and  parameters. 

3.6.5  Definition  and  Execution  of  Critical  Experiments.  Comprehensive  and  often 
standardized  experimental  plans  are  almost  always  executed  to  support  new  material  and 
manufacturing  process  development.  These  may  be  extensive  and  are  intended  to  address  many 
needs,  ranging  from  basic  properties  to  variation  assessments  and  design  or  structures  data 
requirements.  When  ICME  models  and  methods  are  developed,  additional  experimental  data  to 
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support  model  development,  and  eventually  the  requisite  V&V  requirements,  are  also  defined 
and  executed.  In  addition,  experimental  validation  is  essential  for  progression  from  laboratory  to 
pilot  line  to  production  processes.  The  specific  recommendation  made  here,  is  that  an  integrated 
experimental  strategy  should  be  defined  early  in  the  technology  development  process,  and  that 
the  experimental  plan  should  specifically  focus  on  definition  and  execution  of  "critical 
experiments,"  as  a  means  of  reducing  risk  and  demonstrating  capability  for  targeted  conditions, 
geometries,  and  applications.  “Critical  experiments,”  in  this  context,  are  those  experiments  that 
can  reproduce  key  conditions,  test  for  required  or  critical  attributes  in  a  non-ambiguous  manner, 
and  provide  objective,  persuasive  evidence  that  the  manufacturing  technology  has  matured  to  the 
level  expected  for  a  particular  assessment  gate  or  criteria.  They  may  be  part  of  a  standard  test 
protocol  or  plan,  but  are  more  likely  to  be  specially  designed  tests  to  experimentally  assess 
specific  properties,  capability,  or  attributes. 

This  integrated  plan,  in  addition  to  including  (or  referencing)  the  well-established  standard  test 
requirements,  should  also  ensure  the  following  tenets  are  observed  for  the  critical  experiments: 

•  Progressive  use  of  small  to  larger  and  finally  full-scale  or  representative  sizes  be 
employed 

•  Progressive  increase  in  configuration  from  simple  to  application-representative 
geometries  at  full  scale  sizes 

•  Definition  and  execution  of  critical  experiments  and  associated  success  criteria  at  key 
MRL  -  typically  gates  3  or  4,  6,  and  7. 

•  Integration  of  critical  experiments  with  ICME  V&V  requirements,  other  experimental 
plans  including  TRL  required  efforts,  and  with  intended  in-process  quality  monitoring 
and  NDE  developments. 

The  importance  of  carefully  defining  critical  experiments  and  associated  success  criteria,  and  the 
integration  of  these  with  the  overall  experimental  plan,  resulted  in  recommendation  of  this  as  a 
specific  activity. 

3.6.6  Other  Considerations.  One  related  consideration  is  the  likely  need  for  design,  materials, 
and  manufacturing  engineering  skill  development,  specifically  in  the  areas  of  ICME,  process 
modeling,  V&V,  UQ,  and  potentially  other  methods  as  described  in  this  study  task  report.  These 
skills  are  expanding,  but  are  not  yet  broadly  available  in  the  workforce.  Consequently,  training 
and  skill  development  in  specific  topics  as  described  may  be  necessary  for  acceptance  and 
implementation  of  the  study  task  recommendations.  Manufacturing  management  would  likely 
also  benefit  from  training  and  skill  development  in  these  areas,  to  better  understand  the  potential 
benefits,  challenges,  and  resources  needed  to  support  such  activities. 
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4.0  RESULTS  AND  DISCUSSION 


4.1  Concept  of  Systems  Engineering  for  Manufacturing  Development  (SEMT)  and  General 
Alignment  with  MRL 

This  study  has  found  that  the  application  of  system  engineering  (SE)  tools  during  the 
development  of  manufacturing  technology  can  promote  early  identification  of  technical  risks, 
accelerated  development,  and  lead  to  improved,  robust  production  processes  that  yield  higher 
quality  product.  This  report  section  offers  a  general  approach  for  implementing  these  ideas 
within  manufacturing. 

Background  on  SE  implementation 

It  is  widely  recognized  that  engineering  has  profited  from  the  application  of  computational 
models,  particularly  when  used  along  with  integrated  SE  tools.  Benefits  have  included 
accelerated  development  cycles,  reduced  reliance  on  component  testing,  and  improved  product 
performance,  durability,  and  cost.  Similarly,  the  materials  community  is  now  gaining  traction  in 
the  development  and  application  of  models  and  SE  that  support  material  processing  development 
and  prediction  of  material  properties.  However,  the  transition  from  an  experimental/testing¬ 
centric  engineering  workflow  to  one  that  incorporates  both  modeling  and  SE  is  evolutionary, 
with  a  rate  of  advance  that  depends  on  model  maturation  and  the  degree  of  SE  validation.  These 
earlier  examples  provide  some  lessons  that  are  relevant  to  SE  adoption  within  the  manufacturing 
development  community: 

•  The  application  of  modeling  and  SE  should  not  be  prescriptive;  instead  the  IPDT 
should  plan  modeling/SE  efforts  to  meet  program  needs  and  projected  benefits.  A 
prescriptive  approach  for  model/SE  implementation  is  neither  recommended  nor  will 
it  likely  be  successful.  Development  programs  vary  in  how  they  can  apply  and 
benefit  from  modeling  and  SE,  depending  on:  (1)  maturity  of  the  baseline  technology, 

(2)  depth  of  process  knowledge,  (3)  quantity/quality  of  relevant  process  data,  (4) 
availability  of  modeling  tools,  (5)  criticality  of  associated  product  requirements,  and 
(6)  available  funding  and  schedule  -  among  others. 

•  Development  teams  should  identify  the  critical  elements  and  mechanisms  that 
require  physics-based  modeling  using  risk,  PIRT,  and/or  FMECA  analyses  to  guide 
the  team ’s  selection.  These  same  methods  can  be  applied  to  identify  key  data  gaps, 
identification  of  critical  MT  experiments,  and  important  SE  applications  during  the 
MT  development  process 

•  Successful  application  of  SE  requires  a  judicious  balance  between  knowledge 
gained  from  experimental  data  and  physics-based  modeling.  Successful 
implementation  of  modeling  and  SE  tools  depends  strongly  upon  experimental  and 
test  data  to  generate  data-driven  models,  provide  model  inputs,  quantify  constituent 
properties,  establish  model  boundary  conditions,  calibrate  a  model,  quantify 
uncertainty,  or  validate  a  model  or  engineering  analysis. 

Successful  adoption  of  modeling  and  SE  depends  on  trust;  verification  and  validation  are 
paramount.  Changing  or  enhancing  an  engineering  process  -  whether  in  design,  materials,  or 
manufacturing  -  must  overcome  hurdles  (e.g.,  cultural  resistance,  transition  risks  &  costs,  and 
engineer  training).  Management  and  engineers  must  trust  that  SE  implementation  change  leads 
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to  reliable  process  and  product  improvement;  and  achieving  this  trust  is  built  upon  rigorous  and 
complete  verification  and  validation  of  individual  models  and  the  complete  SE-ICME  system. 

Effort  in  Task  3  focused  on  refinement  and  finalization  of  recommended  topics  and  activities  that 
could  accelerate  maturation  and  reduce  risk  for  new  manufacturing  technologies  (MT).  One  key 
challenge,  of  course,  is  how  to  facilitate  implementation?  It  was  believed  essential  that  such 
recommended  activities  be  closely  aligned  with  the  current  MRL/MRA  process,  including  the 
approach  to  sequencing  and  gate  criteria.  It  was  also  believed  essential  that  the  recommended 
activities  be  integrated  into  a  cohesive  package,  rather  than  treated  as  independent,  stand-alone 
topics,  in  order  to  realize  maximum  system  benefit.  The  result  was  to  define  and  propose  an 
additional  “Manufacturing  Thread,”  with  sub-threads  that  contain  the  recommended  topics  and 
activities  identified  in  the  Study  Task.  The  proposed  Systems  Engineering  for  Manufacturing 
Technology  (SEMT)  sub-threads  are  illustrated  in  Table  2,  below,  in  relation  to  the  current  nine 
(9)  manufacturing  threads,  and  to  the  consolidated  listing  of  the  MRL  gates. 

Table  2:  SEMT  Focus  areas  consolidated  as  Sub-threads  in  relation  to  MRL  and  Current 

Manufacturing  Threads 


Manufacturing 

Thread 

MRL  1-4 

MRL  5-6 

MRL  7-8 

MRL  9-10 

Technology 
and  Industrial 

Base 

1.  Technical  Risk  and  Knowledge 
Assessment 

Cost  and 
Funding 

Plan  and  budget  for  acceleration  activities 

Design 

2.  Computational  Modeling  for  Materials  & 
Processes  (ICME)  Development  and 
Implementation 

3.  Verification  &  Validation  and  Tool  Maturity 
Level  (TML) 

4.  Definition  and  Execution  of  Critical  Experiments 

Materials 

Process 

Capability  and 
Control 

5.  Process  Capability,  Control,  and 
Quality  Management 

Quality 

Management 

Manufacturing 

Workforce 

6.  Manufacturing  Management, 

Workforce  Development,  and  Resources 

Facilities 

Identify  and  implement  facilities  and  infrastructure  to  support 
acceleration  processes  and  Digital  Thread 

Manufacturing 

Management 

Manufacturing  and  engineering  management  education  regarding  risk 
ID  and  mitigation,  computational  &  accelerated  maturation  methods 

Legend: 

Bold  Font:  Key  focus  areas.  Standard  Font:  supporting  activities. 
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Figure  7,  below,  is  an  attempt  to  illustrate  these  methods  and  tools  as  an  integrated  approach  to 
accelerate  maturation  of  new  materials  and  manufacturing  technologies.  The  highlighted  boxes 
indicate  recommended  changes  or  additions  to  legacy  practices.  The  Expert  Elicitation  box 
represents  the  Early  Risk  Identification  and  Mitigation  topic,  and  as  shown,  is  intended  to  be 
initiated  early  -  preferably  by  MRL-2,  but  certainly  before  completion  of  the  MRL  3  to  4  gates. 
There  are  several  established  methods  that  can  be  used,  including  PIRT  and  FMECA,  which  are 
regarded  especially  useful  and  were  described  earlier  in  this  report.  The  selection  of  appropriate 
method  and  level  of  detail  will  be  dependent  on  the  application  -  including  the  fidelity  of  model 
or  process  definition  at  the  time.  Methods  such  as  PIRT  could  certainly  be  applied  very  early  - 
even  at  MRL  gates  1  and  2,  and  then  more  rigorous,  detailed  exercises  conducted  as  the 
applications  and  requirements  of  the  technology  become  better  defined.  Also  shown  early  in  the 
process  are  activities  to  define  a  Modeling  Strategy  and  also  a  corresponding  Experimental 
Strategy  -  especially  regarding  definition  of  critical  experiments.  These  strategies  would  be 
expected  to  change  and  mature  over  time,  of  course,  with  more  detail  and  fidelity  developed  as  a 
technology  development  program  progresses. 


Technology  Concept 
Demonstration 

S  "  "N 

Technology/Process 

Feasibility  Initial 
parameters,  controls, 
outputs  defined. 
Applications  and  benefits 

V _ identified _ s 

S  N 

System  or  Product 
Application  -  Initial 
Requirements 

Design,  Performance, 
Cost,  Schedule,  Risks..v 

*  Expert  Elicitation, 

Risk  ID  and  Mitigation 
Plans;  PIRT,  FMEA 

*  Modeling  Strategy 

*  Experimental  Plan 


Manufacturing  Technology: 
Modeling  and  Process  Development 

Technology/Process  Development 

Modeling,  Parameters,  Limits,  Variation, 

>  Controls,  &  Monitoring  Development. 
"/(Material  to  Process  to  Microstructure  to 
Properties  and  Process  Controls...) 

SEMT  Actions 

Risk  ID:  risks  assessed  and  mitigation 
plans  developed  and  implemented 
Integrated  Modeling/Validation: 

ICME,  UQ,  V&V  and  TML  plans 
developed  and  working 
Process  Capability  &  Control:  Model 
updating  initiated,  IPQA™  working,  Key 
Process  Parameters,  Capability  & 
Manufacturing  Variation  defined 
Critical  Experiments:  defined  and 
integrated  in  experimental  plan 


Manufacturing  Technology: 
Rapid  Qualification 

Technology/Process  Qualification 

Parameters,  Limits,  and  Variation 
Validated,  IPQA™  and  NDE  in  place  foF 
technology  and  Process  Qualification  - 
and  Pilot  Line  production 


SEMT  Actions 

Risk  ID:  risks  updated;  mitigation 
plans  executed 

Integrated  Modeling/Validation: 

ICME  models  developed.  UQ,  V&V 
and  TML  plans  executed  &  successful 
Process  Capability  &  Control:  Model 
updating  and  IPQA™  active.  KPPs, 
Capability  &  Mfg.  Variation  verified 
and  active.  Advanced  NDE  in  place 
Critical  Experiments:  and  application 
tests  successfully  executed 


d 


SEMT  Supporting  Infrastructure:  Manufacturing  Management,  Workforce  Development,  and  Resources 


Production 
Implementation^ 
of  Technology  or/ 
Process 


TRL/MRL  ~2  -  4 

Feasibility  Demonstrated 


TRL/MRL  ~4- 6 

Development,  Scale-up,  Demo 


TRL/MRL  ~6- 7 

Rapid  Qualification 


TRL/MRL  ~7- 9 

Production 


Note:  Color  Highlights  Key  Differences 
or  acceleration  from  Legacy  Processes 


Figure  7:  Concept  Illustration  of  Proposed  SEMT  Activities  in  Current  MRL  Process 

Moving  to  the  right  in  Figure  7,  the  SEMT  actions  are  focused  on  supporting  technology 
development,  and  the  expectation  is  that  an  iterative  and  integrated  set  of  activities  as  defined 
under  the  SEMT  thread  would  support  technology  development  -  in  the  range  of  TRL/MRL-4  to 
6,  but  then  continue,  and  support  technology  qualification  and  validation  in  the  range  of  MRL-6 
to  7,  and  beyond.  Efforts  on  ICME  V&V  shift  from  planning  to  execution  as  MRL  status 
progresses  from  MRL-4  to  6  out  to  MRL  -6  to  7  and  beyond.  Also,  as  the  technology  is  matured, 
and  collected  data  increases,  model  updating  methods  can  be  implemented,  continuous  in- 
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process  quality  assurance  and  monitoring  data  can  be  generated  and  analyzed,  and  statistically 
(even  probabilistically)  relevant  data  basis  and  process  limits  established.  Figure  7  is  intended  to 
show  the  close  integration  of  modeling  efforts  with  manufacturing  process  development,  and 
later,  process  and  technology  qualification.  As  illustrated,  it  should  be  an  iterative  and  integrated 
relationship. 

4.2  Proposed  Systems  Engineering  for  Manufacturing  Technology  (SEMT) 

Implementation  of  Systems  Engineering  (SE)  to  support  manufacturing  development  ideally 
should  meld  seamlessly  with  existing  MT  development  processes.  The  format  and  context  of  SE 
insertion  should  be  designed  so  that  it  is  familiar,  concise,  and  easily  understood  by  MT  IPDT 
members  -  yet,  also  supported  by  more  detailed  documentation,  particularly  during  early  stages 
of  deployment. 

The  topic  areas  that  were  identified  and  evaluated  as  described  in  Section  3.6  of  this  report  were 
reviewed  and  consolidated  into  related  activities.  “Expert  Elicitation,  “as  described  in  Section 
3.6.1,  along  with  risk  assessment  and  mitigation  methods  and  non-advocate  review  processes, 
were  included  in  a  single  defined  “sub-thread:”  “Technical  Risk  &  Knowledge  Assessment.” 
Topics  that  were  identified  and  evaluated  in  Section  3.6.2  included  computational  modeling, 
ICME  V&V,  uncertainty  quantification,  and  probabilistic  methods.  These  closely  related  topics 
were  divided  into  two  sub-threads:  “ICME  Development  and  Implementation,”  and  “Verification 
&  Validation  and  Tool  Maturity  Level.”  The  topic  described  as  critical  MT  experiments,  and 
development  of  an  overall  experimental  strategy,  as  described  in  Section  3.6.5,  was  included  as  a 
single  sub-thread,  “Definition  and  Execution  of  Critical  Experiments.”  The  topics  identified  and 
evaluated  as  “in-process  monitoring  and  advanced  inspection  techniques,  from  Section  3.6.4,  and 
“feedback  and  updating  processes,  from  Section  3.6.3,  were  combined  into  a  single  sub-thread: 
“Process  Capability,  Control,  and  Quality  Management.”  Finally,  what  was  described  in  Section 
3.6.6  as  “other  considerations,”  was  developed  into  a  single  sub-thread:  Manufacturing 
Management,  Workforce  Development,  and  Resources.” 

This  grouping  of  recommended  activities  into  defined  “sub-threads”  was  based  on  the  expected 
relationship  of  the  activities  to  one  another,  as  well  as  how  they  align  with  the  current  MRL 
process  and  associated  nine  Manufacturing  Threads.  The  definition  of  these  six  SEMT  sub¬ 
threads,  and  the  way  various  activities  are  included  and  grouped,  could  of  course  be  changed. 

The  natural  insertion  point/vehicle,  that  addresses  these  criteria,  is  the  MRL  Threaded  Matrix. 
This  study  proposes  the  creation  of  one  additional  MRL  thread,  entitled  “System  Engineering  for 
Manufacturing  Technology  (SEMT),”  as  shown  in  Figure  8.  This  new  SE  thread  has  six  sub¬ 
threads,  shown  in  Figure  8,  covering  the  full  spectrum  of  SE  activities  and  MRL  gates  from 
knowledge  assessment,  to  ICME  model  and  data  development,  to  application  of  SE  analysis,  and 
associated  managerial,  workforce,  and  facility  issues. 
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Systems  Engineering 
for  Manufacturing 
Technology 

Xi:  Technical  Risk  &  Knowledge 
Assessment 

X2:  ICME  Development  A 
Implementation 

X3 :  Veri  fi  tati  or  &  Val  i  d  at  ion  &  Too  I 
Maturity  Level 

X4:  Definition  &  Execution  of  Critical 
Experiments 

X5:  Process  Capability,  Control,  & 
Quality  Mgt, 

X6:  Mfg.  Management,  Workforce, 
Development  4  Resources 


Figure  8:  Schematic  illustration  shows  the  proposed  SEMT  thread  and  its  six  Sub-threads 
(light  green  color)  inserted  within  the  MRL  threaded  matrix 

The  content  of  the  six  SEMT  sub-threads  are  listed  and  ordered  by  MRL  level  within  tables 
presented  in  each  of  the  next  six  subsections  of  this  report;  supplemented  by  brief  introductory 
comments.  Because  there  are  numerous  dependencies  among  activities  from  different  sub¬ 
threads,  a  three-step  process  was  used:  (1)  activities  were  listed  for  each  sub-thread;  (2)  each 
listing  was  ordered  to  yield  a  logical  activity  sequence;  and  finally  (3)  MRL  levels  were  assigned 
to  align  the  sub-threads  with  each  other  and  most  importantly  with  program  activities  and 
requirements  of  the  overall  MT  program. 

4.2.1  Sub-thread  1:  Technical  Risk  and  Knowledge  Assessment.  Manufacturing  risk  is  an 
important  focus  of  the  MRL  process;  as  expressed  within  the  MRL  Deskbook:  “The  purpose  of 
an  MRL-based  assessment  of  manufacturing  readiness  is  to  analyze  current  conditions  and  to 
identify  manufacturing  risk  in  order  to  assist  the  program/project  manager  in  creating  a  plan  or 
options  to  reduce  or  remove  risks.”  And  while  the  current  threaded  MRL  Matrix  contains  risk 
assessment  activities,  a  Systems  Engineering  (SE)  sub-thread,  “Technical  Risk  and  Knowledge 
Assessment”  is  proposed  to:  (1)  highlight  the  challenges  of  risk  identification  during 
development  of  novel  MT;  (2)  emphasize  the  importance  of  addressing  risk  assessment  and 
mitigation  earlier  in  development;  and  (3)  recognize  the  beneficial  and  symbiotic  interplay 
between  risk  analysis,  expert  elicitation,  and  SE  tools  for  acquiring  MT  process  knowledge  more 
quickly  and  identifying  development  risk  earlier.  “Expert  Elicitation,  “as  described  in  Section 
3.6.1,  along  with  risk  assessment  and  mitigation  methods  and  non-advocate  review  processes, 
were  included  in  a  single  defined  “sub-thread:”  “Technical  Risk  &  Knowledge  Assessment.” 

Intent  and  Rationale: 

•  Perform  progressive  risk  assessments  within  the  IPDT 
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•  Exercise  expert  elicitation  (EE)  to: 

•  Assess  key  assumptions  within  the  development  strategy 

•  Identify  key  knowledge  gaps  that  must  be  addressed  early 

•  Weave  expert  elicitation  into  the  MRL  &  MRA  processes 

•  Build  complementary  efforts  to  do  DOEs,  models,  UQ  methods  to  fill  knowledge  gaps 

•  Engage  independent  experts  to  probe  scale-up  and  technology  transfer  risks  along  the 
progression  from  laboratory  to  production  implementation. 

Attributes  and  Potential  Benefits: 

•  Identification  of  risk  drivers  and  pitfalls  earlier 

•  Use  of  expert  elicitation  processes  such  as  PIRT  and  FMECA 

•  Early  &  more  accurate  program  development  decision-making. 

Almost  all  MT  development  programs  do  encounter  risks,  which  while  not  necessarily 
debilitating,  can  disrupt  MT  development  progress,  by  requiring  expanded  and  extended 
research,  modifying  or  adding  a  process  step(s),  or  causing  retrenchment  from  original  program 
goals,  payoff,  and  reduction  of  implementation  opportunities.  MT  development  programs 
succeed  when  developers  proactively  include  research  and  testing  efforts  to  uncover  and  resolve 
risks  quickly;  instead  of  focusing  solely  on  research  directed  to  prove  MT  benefits.  Indeed,  the 
ability  of  an  MT  IPDT  to  perceive  risk  depends  on  the  knowledge  and  breadth  of  experience  of 
the  IPDT  team  members.  Accelerating  identification  of  “low  probability  of  occurrence”  risks  is 
aided  by  the  following  strategies: 

•  Involving  independent  experts  to  augment  the  IPDT  depth  and  breadth  of  knowledge; 

•  Accelerating  MT  experimental  research,  adapting  and  applying  Integrated  Computational 
Materials  Science  and  Engineering  (ICMSE),  and  implementing  uncertainty 
quantification  (UQ)  for  lesser  known  MT  sub-processes;  and 

•  Adding  focus  on  increasing  the  robustness  for  sub-process(es)  that  exhibit  high 
variability  during  early  MT  experiments  and  trials. 

This  sub-thread  focuses  on  Expert  Elicitation,  which  entails  methods  to  expand  the  knowledge 
and  to  sharpen  the  technical  insight  of  the  team  assessing  the  MT  development.  Expert 
elicitation  (EE)  is  typically  carried  out  using  standard  team  processes,  such  as  the  Phenomena 
Identification  and  Ranking  Technique  (PIRT)  and  the  Failure  Modes,  Effects,  and  Criticality 
Analysis  (FEMCA)  by  an  independent  team  composed  of  experts  from  within  or  outside  the 
companies  engaged  in  the  MT  development  effort;  see  further  details  in  the  appendix. 

Implementation  of  EE  can  have  far  reaching  impact  on  MT  development  program  success, 
particularly  if  it  is  initiated  in  the  early  stages  of  development.  EE  benefits  extend,  either  directly 
or  indirectly,  to  the  other  systems  engineering  strategies  and  tools  and  these  benefits  accrue 
through  the  enlightening  effect  of  knowledge  gained  to  inform  downstream  decisions. 

Development  of  manufacturing  technology  is  highly  dependent  on  both  engineering  knowledge 
and  judgment.  Mathematical  and  data-driven  models  as  well  systems  engineering  tools  can  have 
a  powerful  impact  on  the  rate  of  development  progress  and  success  -  but  only  after 
manufacturing  engineers  have  defined  the  MT  project  and  requirements,  identified  the 
manufacturing  process,  uncovered  critical  process  mechanisms  and  risks,  and  selected  the 
modeling,  critical  data,  and  experimentation  that  collectively  can  accelerate  and  enhance  MT 
development. 
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The  sub-thread  addresses  the  application  of  engineering  knowledge  to: 

•  Identify,  assess  program  risk  and  define  potential  mitigation  activities, 

•  Identify  MT  knowledge  gaps  and  define  engineering  analysis  and  experimentation  to  fill 
the  gaps,  and 

•  Isolate  potential  process  failure  mechanisms,  and  provide  remedial  and  abatement 
countermeasures . 

For  this  SEMT  sub-thread,  recommended  activities  and  their  alignment  with  MRL  gates  are 
presented  in  Table  3,  below. 


Table  3:  Technical  Risk  and  Knowledge  Assessment  Sub-thread  and  MRL  Alignment 


# 

MRL 

Level 

Risk  and  Knowledge  Assessment 

1 

2 

Initial  risk  assessment  completed  by  Manufacturing  Technology  IPDT. 
Assessment  of  TRL  plan  (the  TDP)  completed.  Integration  of  MRL  and  TRL 
activities  planned. 

2 

3 

Manufacturing  technology  (MT)  Expert  Elicitation  panel  selected  and 
engaged  for  subsequent  PIRT  and  FMECA  analyses. 

3 

3 

Rules  of  engagement  defined  for  Expert  Elicitation  PIRT  analysis  and 
transmitted  along  with  relevant  MT  and  TRL  documentation  to  Expert 
Elicitation  panel. 

4 

3 

Expert  Elicitation  PIRT  analysis  completed  and  presented  to  MT  IPDT. 

Results  communicated  to  TRL  team. 

5 

4 

Component  and  Material  requirements  have  been  documented  and 
conveyed  to  Expert  Elicitation  panel  for  subsequent  risk  and  knowledge 
assessments. 

6 

4 

Interim  MRL-4  risk  assessment  completed  by  Manufacturing  Technology 

IPDT  with  support  from  Expert  Elicitation  panel  and  PIRT  results.  Risk 
mitigation  plan  updated  and  integrated  with  TRL  activities. 

7 

5 

Rules  of  engagement  defined  for  Expert  Elicitation  FMECA  analysis  and 
transmitted  along  with  relevant  MT  and  TRL  development  documentation  to 
Expert  Elicitation  panel. 

8 

5 

Expert  Elicitation  FMECA  analysis  completed  and  presented  to  MT  IPDT. 

9 

6 

Interim  risk  assessment  updated  by  MT  IPDT  with  support  from  Expert 
Elicitation  panel  and  FMECA  results.  Risk  mitigation  plan  updated. 

The  tools  involve  standard  risk  analysis  and  expert  elicitation  (EE)  methods  as  exercised  by  the 
MT  IDPT  or  an  independent  panel  composed  of  external  domain  experts  from  government  labs, 
academia,  or  industry.  In  addition,  the  IPDT  may  benefit  by  replacing  critical  EE  exercises  with 
a  formal  Non- Advocate  Review  (NAR). 


4.2.2  Sub-thread  2:  Integrated  Computational  Materials  Engineering  (ICME).  A  fully 
mature  ICME  system  includes  (1)  mathematical  models  covering  physically-based  and  data- 
derived  knowledge;  (2)  databases  for  material  properties,  physical  constants,  &  experimental 
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data;  (3)  SE  tools  for  DOE,  sensitivity,  optimization,  and  robust  design  analysis.  Collectively 
these  represent  the  sources  of  domain  knowledge;  whereas  the  third  provides  the  means  and 
methods  to  exercise  this  knowledge  to  improve  manufacturing  processes.  ICME  and 
computational  modeling  in  general  were  identified  and  evaluated  in  Section  3.6.2  and  are  now 
included  in  the  2nd  sub-thread:  “ICME  Development  and  Implementation.” 

Intent  and  Rationale: 

•  Application  of  ICME  models  that  incorporate  material  science  mechanisms  and 
associated  process  physics  can  significantly  accelerate  knowledge  acquisition: 

•  Identify  and  interpret  process  anomalies  earlier 

•  Improve  manufacturing  processes  more  quickly 

•  Integrate  ICME  and  Digital  Thread  for  better  development  analysis 

•  Apply  System  Engineering  tools  to  reduce  development  iterations  leading  to  more  robust 
controlled  manufacturing  processes 

Attributes  and  Potential  Benefits: 

•  Organized  approach  that  yields  component  improvement  more  quickly,  at  lower  cost,  and 
lower  risk 

•  Promote  reuse  of  models  &  knowledge  for  down-stream  improvement  and  developments 

•  Reduce  risk  of  development  iterations,  and  facilitate  faster,  less  costly  response  to  any 
system  or  design  iterations 

Ideally,  an  ICME  system  will  encompass  comprehensive  quantitative  modeling  of  a  material  - 
whether  metal  alloy  or  composite  -  including  its  composition,  processing,  resulting 
microstructure,  and  finally  the  mechanical  behavior  and  properties  of  the  material  relevant  to  the 
application  of  interest.  While  there  are  many  challenges  in  the  development  of  the  integrated  and 
linked  models  that  comprise  an  ICME  system,  the  capability  has  been  demonstrated  in  many 
programs,  and  utilization  of  ICME  is  increasing  rapidly.  The  computational  models  that 
comprise  an  ICME  system,  and  the  supporting  data  used  to  generate  the  models,  will  facilitate 
much  more  direct  integration  of  materials  and  manufacturing  processes  with  the  design, 
structural  analysis,  life  prediction,  and  quality  disciplines.  Development  and  implementation  of 
comprehensive,  validated  ICME  methods  and  models  will  reduce  dependence  on  lengthy, 
expensive  test  programs,  reduce  the  number  of  development  iterations,  accelerate  process 
optimization,  enable  tailoring  to  design  and  structural  requirements,  and  guide  informed  process 
control  and  accelerated  process  qualification. 

For  this  SEMT  sub-thread,  recommended  activities  and  their  alignment  with  MRL  gates  are 
presented  in  Table  4,  below. 
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Table  4:  ICME  Sub-thread  and  MRL  Alignment 


# 

MRL 

Level 

Integrated  Computational  Materials  Engineering  (ICME) 

1 

2 

Notional  ICME  system  outlined  and  integrated  with  TRL  ICME  plan,  including 
expected  key  models  and  tools.  Significant  modeling  gaps  identified.  Initial 
ICME  modeling  and  V&V  strategy  defined. 

2 

3 

High-level  ICME  development  plan  established  and  budgetary  requirements 
determined  by  IPDT. 

3 

3 

Initial  ICME  modeling  system  identified,  diagrammed  with  informational  flows, 
and  technical  challenges  identified  and  integrated  with  TRL  activities. 

4 

3 

MT  IPDT  has  assembled  ICME  system  development  team;  and  initiated 
model  development  to  achieve  MRL-4  requirements. 

5 

3 

Development  priorities  for  new  ICME  models  established  based  on  PIRT 
results  and  material/component  requirements. 

6 

4 

ICME  team  has  coordinated  with  MT  IPDT  and  TRL  team  to  schedule  MT 
experiments  that  support  ICME  development. 

7 

4 

ICME  has  identified  and  procured  "system  engineering"  software  (e.g., 
Phoenix's  Model  Center  or  iSIGHT)  to  conduct  analytical  DOE,  sensitivity, 
response  surface,  optimization  and  robust  manufacturing  analyses. 

8 

4 

ICME  team  has  developed  ICME  system  that  satisfies  MRL-4  requirements. 

9 

5 

ICMSE  "Digital  Thread"  data,  data  base,  infrastructure,  and  technologies,  that 
support  ICME  development,  V&V,  and  UQ  analysis,  have  been  identified, 
designed  and  implemented. 

10 

5 

ICSME  team  has  conducted  a  Monte  Carlo  analysis  to  synthesize  MT  system 
variation  based  on  process  input  variables  uncertainty/variation. 

11 

6 

Development  priorities  and  new  ICME  models  have  been  updated  based  on 
FMECA  results  to  achieve  MRL-7  requirements. 

12 

7 

ICME  team  has  developed  ICME  system  that  satisfies  MRL-7  requirements. 

4.2.3  Sub-thread  3:  Verification  and  Validation  and  Tool  Maturity  Level.  Verification, 
validation  and  Tool  Maturity  Level  (TML)  assessment  is  critical  for  building  confidence  that  an 
SEMT  analysis  has  acceptable  accuracy  and  uncertainty  bounds.  Although  the  activities  for  this 
V&V/TML  sub-thread  are  intimately  linked  to  the  second  sub-thread  on  ICME,  it  has  been 
allocated  a  separate  sub-thread  for  clarity,  to  emphasize  its  importance,  and  to  acknowledge  that 
V&V  exercises  requires  independent  data  to  validate  both  ICME  model  and  system  level 
predictions.  Also  included  in  this  sub-thread  is  the  topic  of  uncertainty  quantification  (UQ),  as 
an  essential  component  of  ICME  V&V  assessment.  In  addition,  UQ  is  an  essential  part  of 
assessing  manufacturing  variation  -  especially  process  variation  -  which  is  addressed  in  another 
sub-thread.  These  topics  were  identified  and  evaluated  as  described  in  Section  3.6.2,  and  a 
number  of  references  were  provided  in  that  section. 

Intent  and  Rationale: 

•  Successful  &  trustworthy  application  of  ICME  and  Systems  Engineering  depends  on: 

•  Verifying  that  ICME  models  are  solving  equations  correctly 
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•  Validating  that  ICME  modeling  results  match  the  real-world  application 

•  Quantifying  uncertainty  (UQ)  for  both  program  data  and  simulation  results 

•  Ensuring  that  current  modeling  maturity  meets  application  requirements 

•  Follow  the  V&V  and  Tool  Maturity  guidelines  developed  by  AFRL  &  MAI,  or 
equivalent  process 

Attributes  and  Potential  Benefits: 

•  Builds  trust  in  ICME  &  Systems  Engineering  within  the  IPDT  and  with  the  customer 

•  Ensures  V&V  activity  is  appropriate  and  sufficient  for  the  system  or  product  application 

•  Provides  quantitative  understanding  of  uncertainty  that  promotes  improved  development 
decision  making. 

Recommended  activities  and  their  alignment  with  MRL  gates  are  presented  in  Table  5,  below. 

Table  5:  V&V  and  TML  Sub-thread  and  MRL  Alignment 


# 

MRL 

Level 

Verification  &  Validation  and  Tool  Maturity  Level 

1 

3 

Initial  planning  V&V  Checklist  completed  and  integrated  with  TRL  plans  for 
ICME  development  and  associated  V&V. 

2 

3 

Tool  Maturity  Levels  (TML)  requirements  assessed  and  coordinated  with  TRL 
plans  to  guide  planning  of  V&V  activities  and  associated  budgetary 
requirements. 

3 

4 

ICME  team  has  identified  initial  TMLs  for  individual  models  and  tools. 

4 

4 

ICME  team  has  coordinated  with  MT  IPDT  and  TRL  team  to  define  and 
schedule  MT  experiments  that  support  ICME  V&V. 

5 

4 

ICME  team  has  applied  initial  ICME  system  to  identify  the  most  influential 
input  and  internal  parameters  governing  ICME  simulation  and  MT  system 
output  uncertainty. 

6 

4 

ICME  team  has  coordinated  with  MT  IPDT  and  TRL  team  to  acquire  MT 
development  data  to  quantify  uncertainty  for  highly  sensitive  MT  process  input 
variables. 

7 

4 

ICME  has  completed  a  MRL-4  update  of  the  V&V  checklists,  TML 
assessment,  V&V  Dashboard  and  concurrence  that  MRL-4  V&V  requirement 
have  been  met. 

8 

5 

ICMSE  team  has  designed  verification  requirements  for  individual  models  of 
the  ICME  system. 

9 

5 

ICME  team  has  confirmed  that  verification  requirements  of  all  models  in  the 
ICME  system  have  been  achieved. 

10 

5 

ICME  team  has  designed  validation  experiments  to  validate  individual  models 
in  the  ICME  system. 

11 

5 

ICME  team  has  designed  validation  experiments  to  validate  the  fully 
integrated  ICME  system. 

12 

6 

ICSME  team  has  completed  a  MRL-7  update  of  the  V&V  checklists,  TML 
assessment,  V&V  Dashboard,  and  concurrence  that  MRL-7  V&V  requirement 
have  been  met. 
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For  an  ICME  application  to  be  successful  and  to  credibly  support  decision-making,  rigorous 
verification  and  validation  processes,  including  UQ,  are  essential.  This  is  truly  a  critical 
requirement  to  ensure  that  program  management,  especially  the  chief  engineers  for  the  end 
product  or  application,  and  regulatory  authorities,  have  confidence  in  the  fidelity,  maturity,  and 
validation  of  model-based  decisions. 

4.2.4  Sub-thread  4:  Definition  and  Execution  of  Critical  Experiments.  Experimental  plans 
are  always  generated  to  support  new  material  and  manufacturing  process  development.  These 
may  be  extensive  and  are  intended  to  address  many  needs,  ranging  from  basic  properties  to 
variation  assessments,  ICME  model  development,  and  design  or  structures  data  requirements. 
The  specific  recommendation  made  here,  is  that  an  integrated  experimental  strategy  should  be 
defined  early  in  the  technology  development  process,  and  that  the  experimental  plan  should 
specifically  focus  on  definition  and  execution  of  "critical  experiments,"  as  a  means  of  reducing 
risk  and  demonstrating  capability  for  targeted  conditions,  geometries,  and  applications.  “Critical 
experiments,”  in  this  context,  are  those  experiments  that  can  reproduce  key  conditions,  test  for 
required  or  critical  attributes  in  a  non-ambiguous  manner,  and  provide  objective,  persuasive 
evidence  that  the  manufacturing  technology  has  matured  to  the  level  expected  for  a  particular 
assessment  gate  or  criteria.  They  may  be  part  of  a  standard  test  protocol  or  plan,  but  more  likely 
they  may  be  a  specially  designed  test  to  experimentally  assess  specific  properties,  capability,  or 
attributes.  The  topic  described  as  critical  experiments,  including  development  of  an  overall 
experimental  strategy,  was  described  in  Section  3.6.5,  and  is  included  as  a  single  sub-thread, 
“Definition  and  Execution  of  Critical  Experiments.” 

Intent  and  Rationale: 

•  Technology  development  process  includes  extensive  experimental  effort 

•  ICME,  associated  V&Y,  and  key  process  parameter  monitoring  will  add  significant 
experimental  knowledge  and  should  be  integrated 

•  Critical  experiments  should  be  defined  in  advance  for  key  milestones  or  gates 

•  A  comprehensive  experimental  strategy  should  be  developed  early 

Attributes  and  Potential  Benefits: 

•  Comprehensive  experimental  strategy  will  help  integrate  TRL  and  MRL  efforts 

•  Definition  of  critical  experiments  will  guide  effort  and  provide  objective  measures  of 
success  -  building  early  confidence  and  accelerating  maturity 

The  importance  of  carefully  defining  critical  experiments  and  associated  success  criteria,  and  the 
integration  of  these  with  the  overall  experimental  plan,  resulted  in  recommending  this  as  a 
specific  sub-thread  for  "Accelerated  Manufacturing  Technology  Development.”  Recommended 
activities  and  their  alignment  with  MRL  gates  are  presented  in  Table  6,  below.  Note  that  an 
initial  experimental  plan  with  notional  description  of  critical  experiments  should  be  initiated 
early:  here  it  is  recommended  at  MRL-2.  Subsequent  activities  are  expressed  as  recommended 
gate  exit  criteria  for  key  MRL  gates. 
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Table  6:  Critical  MT  Experiments  Sub-thread  and  MRL  Alignment 


# 

MRL 

Level 

Critical  MT  Experiments 

1 

2 

Notional  experimental  plan  defined,  including  progression  from  small,  simple 
geometry  coupons  through  larger,  complex  geometry  sub-elements  and  full- 
scale  articles.  Integrated  with  TRL  plans  as  appropriate. 

2 

3 

High-level  experiment  plan  established  and  budgetary  requirements 
determined  by  IPDT. 

3 

3 

Process  feasibility  demonstration  experiments  defined  and  successfully 
conducted. 

4 

4 

Process  application  experiments  defined  and  successfully  completed 
suitable  for  MRL4  level  of  application  knowledge 

5 

5 

Critical  experiments  for  process  demonstrations  and  success  criteria  for 
specific  applications  defined  in  detail  for  MRL6,  7,  and  8  gate  requirements. 

6 

6 

Critical  experiments  for  representative  scale-up  size  and  geometry  features 
successfully  completed. 

7 

7 

Critical  experiments  including  use  of  "copy  exactly"  processing  (if 
appropriate)  successfully  completed  on  representative  size  and  geometry 
article(s). 

8 

8 

Critical  experiments  including  use  of  "copy  exactly"  processing  (if 
appropriate)  and  intended  IPQA  methods  successfully  completed  on 
detailed  design-representative  article 

4.2.5  Sub-thread  5:  Process  Capability,  Control,  &  Quality  Management.  In  the  current 
MRL  construct,  there  are  separate  threads  for  Process  Capability  and  Control,  and  for  Quality 
Management,  each  having  three  sub-threads.  In  practice  they  are,  of  course,  closely  related  and 
integrated.  The  topics  identified  and  evaluated  as  “in-process  monitoring  and  advanced 
inspection  techniques,  from  Section  3.6.4,  and  “feedback  and  updating  processes,  from  Section 
3.6.3,  were  combined  into  a  single  sub-thread:  “Process  Capability,  Control,  and  Quality 
Management.”  These  were  described  extensively  in  Sections  3.6.3  and  3.6.4,  so  descriptions  are 
not  repeated  here. 

Intent  and  Rationale: 

•  Integrate  items  from  several  related  Manufacturing  Threads  into  a  common  strategy  for 
process  measurement,  control  and  updating 

•  Define  key  process  parameters  and  relate  them  to  ICME,  process  modeling,  and 
technology  risk  assessment 

•  Formalize  a  process-data  updating  method  (Bayesian  or  alternative) 

•  Integrate  in-process  monitoring  and  advanced  NDE  methods 

Attributes  and  Potential  Benefits: 

•  Integrates  all  available  process  data  for  earlier  maturation  and  confidence 

•  Integration  of  model-based  knowledge  with  data  to  achieve  more  confidence  earlier 
(target  “six-sigma”  or  equivalent  before  MRL  8) 

•  Provide  formal  basis  for  continuous  improvement  vs.  “frozen  process”  approach  to 
maturation  and  improvement  of  new  materials  and  process  technology. 
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Experimental  data  is  continuously  generated  as  a  normal  part  of  Manufacturing  Technology  and 
supporting  ICME  developments.  This  experimental  data  is  important  for  demonstration  and 
quantification  of  new  process  capability  and  output.  It  is  also  essential  to  define  process  limits 
and  controls,  to  validate  sensitivity  and  effect  of  key  process  parameters,  and  ultimately  to 
validate  and  support  process  qualification  for  intended  applications.  This  data  is  also  important 
for  another  purpose:  a  rigorous  and  systematic  updating  process  to  refine  and  improve  ICME 
models  and  parameters,  including  their  variation,  which  are  deemed  important  to  represent  the 
materials  and  manufacturing  processes  of  interest.  What  is  recommended  as  a  result  of  this  study 
task  is  systematic  and  disciplined  use  of  updating  and  refinement  methods  for  both  the 
computational  modeling  efforts  and  the  manufacturing  processes  themselves,  beginning  in  the 
MRL  -4  to  5  time  period  and  continuing  thereafter. 

In-process  quality  monitoring,  and  advanced  inspection  capability,  offer  independent  methods  to 
validate  process  output,  ensure  processes  are  operating  within  defined  limits,  and  even  provide 
potential  for  closed-loop  control  of  key  process  parameters  that  was  not  previously  possible. 
While  such  capability  would  be  very  beneficial  as  independent  elements,  they  can  be  especially 
effective  when  coupled  with  ICME-enabled  material  and  process  modeling,  and  uncertainty 
quantification  of  key  model  parameters  and  outputs.  These  methods  offer  potential  to  develop 
significant  understanding  of  process  capability,  performance,  and  limits  so  that  qualification  for 
various  applications  can  be  greatly  accelerated,  and  implementation  risks  greatly  reduced. 

This  sub-thread  describes  activities  which  would  be  essential  to  integrate  the  ICME  modeling 
efforts,  as  well  as  advanced  process  monitoring  and  NDE  capability  with  traditional  quality 
control  and  management,  including  statistical  process  control  methods.  The  recommended 
activities  are  believed  consistent  with  the  AFRL  Digital  Thread  initiative,  and  could  be  used  to 
help  define  application-specific  requirements  for  that  initiative  as  part  of  manufacturing 
technology  planning.  The  combination  of  these  activities  makes  up  the  SEMT  recommended 
sub-thread  for  Process  Capability,  Control,  and  Quality  Management.  The  recommended  timing 
and  sequence  is  shown  in  relation  to  the  MRL  levels  and  Manufacturing  Threads  in  Table  7. 
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Table  7:  Process  Capability,  Control  and  Quality  Sub-thread  and  MRL  Alignment 


# 

MRL 

Level 

Process  Capability,  Control,  and  Quality  Management 

1 

4 

ICME  Digital  Thread  amended  and  appended  to  support  MT  quality  system. 

2 

5 

ICME  sensitivity  study  results  and  initial  QC  data  applied  to  guide  QC  data 
acquisition  plan. 

3 

5 

ICME  system  engineering  tools  applied  to  examine  MT  system  dynamics, 
robust  MT  system  operation,  and  response  surfaces  to  generate  information  to 
aid  MT  control  system  development. 

4 

5 

Key  process  and  material  parameters  identified  for  active  process  monitoring 
and  control.  Sensor,  monitoring  and  key  data  requirements  identified,  and 
integrated  with  ICME  models  and  data  feedback  requirements. 

5 

6 

ICMSE  team  has  developed  an  approach  and  software  to  implement  Bayesian 
or  equivalent  updating  to  refine  assessment  of  QC,  ICME,  and  quality  data. 

6 

6 

Key  process  and  material  monitoring  and  control  sensors  and  data 
implemented  into  development  process.  Data  and  process  output  integrated 
with  ICME  models.  Continuous  or  periodic  process  update  plans  developed, 
using  data  feedback  and  ICME  models. 

7 

7 

Integration  of  key  process  data  and  ICME  models  with  UQ  and  variation 
assessments  completed.  Level  of  Statistical  Process  Control  determined,  and 
requirements  to  achieve  Six  Sigma  equivalent  control  have  been  defined. 

8 

8 

Six-Sigma  or  equivalent  certified  process  control  achieved  -  and  integrated  with 
ICME  models  and  in-process  monitoring  approaches  for  periodic  or  continuous 
feedback,  sustainment,  and  improvement.  Process  variation  monitored  and 
controlled. 

4.2.6  Sub-thread  6:  Manufacturing  Management,  Workforce  Development,  and  Resources. 

Manufacturing  Workforce,  Facilities,  and  Manufacturing  Management  represent  three  separate 
threads  in  the  current  MRL  construct.  There  are  additional  activities  that  are  recommended  in 
these  areas  as  part  of  the  SEMT  thread.  What  was  described  in  Section  3.6.6  as  “other 
considerations,”  was  developed  into  a  single  sub-thread:  Manufacturing  Management, 
Workforce  Development,  and  Resources.” 

Intent  and  Rationale: 

•  Integrate  manufacturing  management  and  resource  items  from  several  related,  existing 
Manufacturing  Threads,  to  address  SEMT  requirements 

•  Address  management  education  and  advocacy  building  for  ICME  and  related  SEMT 
precepts 

•  Ensure  manufacturing  and  engineering  workforce  skill  development  in  SEMT  methods 
and  tools 

•  Identify  facility  and  resource  needs  for  SEMT  success 
Attributes  and  Potential  Benefits: 

•  Education  and  skill  development  in  computational  models,  methods  and  associated  UQ 
and  V&V  activities 
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•  Provide  skills  and  methods  to  better  integrate  manufacturing  with  materials  and 
engineering  disciplines 

•  Education  leads  to  advocacy  -  necessary  to  improve  the  current  paradigm:  iterative, 
empirical,  frozen  process,  “freeze  and  characterize.” 

It  is  believed  that  the  most  important  activity  here  will  be  the  need  for  materials  and 
manufacturing  engineering  employee  skill  development,  specifically  in  the  areas  of  ICME, 
process  modeling,  V&V,  UQ,  and  potentially  other  methods  as  described  in  this  study  task 
report.  These  skills  are  expanding,  but  are  not  yet  broadly  available  in  the  workforce. 
Consequently,  training  and  skill  development  in  specific  topics  as  described  may  be  necessary 
for  acceptance  and  successful  implementation  of  the  study  task  recommendations.  Education  and 
commitment  of  manufacturing  management  will  also  be  essential:  first  to  understand  and 
advocate  for  the  recommended  SEMT  activities,  and  then  to  ensure  early  and  continuous 
commitment  of  required  resources.  Increased  use  of  computational  modeling,  integration  of 
process  development  and  process-monitoring  data,  NDE  results,  automated  data  collection  and 
analysis  for  feedback  -  will  necessitate  investment  in  equipment,  data  bases,  personnel,  and 
software.  Collectively,  a  commitment  to  SEMT  activities  offers  enormous  potential  for  benefits 
in  time,  cost,  and  risk  for  MT  development  and  implementation:  but  achieving  these  benefits  will 
require  that  manufacturing  management  understand  and  support  the  infrastructure  and  workforce 
requirements.  The  timing  and  sequence  of  recommended  activities  are  shown  in  relation  to  the 
MRL  levels  in  Table  8. 

Table  8:  Manufacturing  Management,  Workforce  Development,  and  Resources  Sub-thread 

and  MRL  Alignment 


# 

MRL 

Level 

Manufacturing  Management,  Workforce  Development,  and 

Resources 

1 

3 

MT  IPDT  has  briefed  management  on  the  planned  system  engineering  approach, 
integration  with  the  related  TRL  activities,  and  expected  quality,  acceleration,  and  risk 
reduction  benefits. 

2 

3 

MT  IPDT  has  provided  management  with  a  plan  and  cost  analysis  for  the  MT  system 
engineering  effort. 

3 

3 

MT  management  has  committed  sufficient  funds  and  internal  workforce  resources  to 
execute  systems  engineering  acceleration  activities. 

4 

4 

MT  IPDT  has  provided  to  management  a  manpower  plan  that  distinguishes  between 
in-house  and  out-sourced  manpower  (and  associated  rationale  and  costs). 

5 

4 

MT  management  has  committed  to  provide  resources  that  support  computer, 
database,  and  computational  software  associated  with  ICME,  V&V,  and  Digital 

Thread  activities. 

6 

4 

MT  IPDT  has  provided  a  summary  of  workforce  expertise  needs  (i.e.,  in  ICME, 
process  modeling,  V&V,  UQ,  and  leftward  implementation  of  system  engineering 
technologies)  to  achieve  a  long-term  MT  improvement  and  subsequent  sustainment. 

7 

4 

MT  management  has  committed  to  provide  sufficient  MT  development  facilities  (or  a 
fair  time  share)  to  support  ICME  and  associated  systems  engineering  development 
activities. 

8 

5 

Workforce  development  activities  for  training  in  ICME,  UQ,  V&V,  and  In-process 
monitoring  methods  planned  and  initiated. 
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Note  that  significant  activities  related  to  SEMT  align  with  MRL-3,  4,  and  5.  These  activities  are 
not  intended  to  supersede  any  of  the  similar  activities  which  appear  in  related  Manufacturing 
Threads  under  the  current  MRL  construct,  but  rather  to  initiate  them  earlier  and  to  ensure  focus 
on  the  requirements  and  recommendations  found  in  the  SEMT  Thread. 

4.3  Alignment  of  SEMT  with  the  MRL  Process  and  Manufacturing  Threads 

The  proposed  new  Manufacturing  Thread:  “Systems  Engineering  for  Manufacturing 
Technology”  has  been  defined,  along  with  the  six  sub-threads  that  contain  recommended 
activities  and  their  proposed  sequence.  These  were  described  in  some  detail  in  the  previous 
sections  of  this  report,  and  are  presented  in  tabular  form  in  Tables  3  through  8,  along  with 
proposed  alignment  with  MRL  gates.  The  intent  was  to  ensure  close  alignment  and  compatibility 
with  the  current  MRL/MRA  process. 

It  is  recognized  that  some  of  the  activities  recommended  in  the  SEMT  thread  are  represented  in 
other,  existing  threads.  In  these  cases,  it  is  believed  that  the  SEMT  thread  advocates  earlier 
initiation  of  specific  activity,  and  in  most  cases  provides  a  more  defined  set  of  activities  and  how 
these  should  be  integrated  with  other  manufacturing  threads.  References,  historical  perspective, 
and  some  examples  were  provided  in  Section  3.6  of  this  report. 

Illustrating  the  entire  MRL  gate  structure,  with  all  nine  current  threads,  is  difficult  without 
reducing  detailed  entries  to  unreadable  size.  It  is,  however,  useful  to  show  the  alignment  of  the 
proposed  SEMT  thread,  and  its  six  proposed  sub-threads,  with  the  MRL  gates  and  the  important 
corresponding  DoD  Acquisition  Milestones:  A,  B,  and  C.  This  alignment  is  shown  in  Figure  9, 
below.  Please  note: 

1.  The  “Systems  Engineering  for  Manufacturing  Technology  (SEMT)”  thread  spans  the 
entire  MRL  gate  regime,  although  gate  criteria  have  not  yet  been  defined  for  some  MRL 
levels. 

2.  Six  sub-threads  comprise  the  recommended  activities  and  have  been  individually  aligned 
with  MRL  gates. 

3.  Significant  activity  is  recommended  early  in  the  MRL  process:  defined  actions  are 
included  under  MRL  2  and  3,  and  most  activity  is  defined  prior  to  Milestone  B,  at 
conclusion  of  MRL  6. 

The  details  of  the  proposed  activities,  and  their  alignment  with  the  MRL  gates,  represent  an 
initial  proposal  based  on  this  brief  Study  Task  and  the  feedback  which  was  received  during  its 
execution.  A  thorough  review  and,  ideally,  some  trial  application,  would  undoubtedly  result  in 
some  revision  and  improvement. 
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DoD  Manufacturing  Readiness  Levels  (MRLsi 

Acquisition  Phase 

Pie  Materiel  Solution  Analgsis  [Pie  MSA) 

Materiel  Solution  Analysts 
(MSA)  A 

Ttchnologj  Development  [TD] 

Engineering  &  Mfg  Development  (EMD) 

Low  Ratf  Initial  Production  1 

(LRIP)  1 

Full  Rale  Production 

(FRP) 

~ 

Ttchmcil  Reviews 

ASR  ^ 

SRRfSFR 

CDR 

PRRISYR  PCA 

Thu-ad 

Sub-Thread 

MRU 

MRL  2 

MRL  3 

MRL4 

MRL  5 

MRL  £  || 

MR.  1 

MRL  8 

MRL$ 

MRL  10 

Technoloqf 

Maturity 

Should  be  assessed  at  TRI  2. 

Should  be  assessed  M  TRL  3. 

Should  be  assessed  it  TRL  4, 

Should  be  assessed  at  TRL  T, 

Should  be  assessed  at  TRL  6. 

Should  be  assessed  at  TRL  I 

Should  be  assessed  at  TRI  T 

5h®*id  be  a:  stssed  at  TRt  3. 

Should  be  assessed  at  TRL  J. 

X.  Systems  Engineering  for  Accelerating  Manufacturing  Technology  Development 

XI  - Technical 
Risk  k 
Knowledge 
Assessment 

Im*  a!  tick  assessment  completed  by 
Manufae  'wing  T«  Lnolcgy  IPOT. 

Asse:  ;im*  ©f  TW  pfci  (» he  TOP) 
completed.  Intonation  or  MRL  and 

TRI  activities  planned 

MT  Expert  Elicitation  (EE)  panel 
seketed/eofttraeted  for  PIRT  and 
FMECA  analyses,  EE  PfflT  rule;  Of 
engagement  defined  4  sent  with  key 

MT  andTRL  document;  to  EE  paivd 

EE  PIRT  analysis  Completed  and 
pmHtdlHM  to  MT  IPCT  S  TRI. 

tOHR. 

Requirements  documented  and 
conveyed  to  EE  panel  for  risk  4  PIRT 
assessments.  Interim  MRL-i  risk 
assessment  completed  by  MT  IPOT 
with  input  from  EE  panels,  Risk 
mitigation  plan  updated  and  integr  ated 
with  TRL  activities. 

Rule;  of  engagement  defined  for  EE 
FMECA  analysis  and  transmitted  with 
relevant  MT  and  TRL  development 
documentation  to  EE  panel.  EE 

FMECA  analysis  completed  and 
presented  to  MT  IPDT. 

Interim  risk  assessment  upd  a  ted  by  MT 
IPDT  with  support  f  rom  EE  panel  and 
FMECA  results,  ffiskmitigauompfeo 
updated. 

X2-ICME 

Development 

t 

Implement  atio 
n 

Motion  V ICME  system  ou»  Ined  and 
migrated  nHiMRl  CMC  plan, 
racking  expected  key  r  >dd;  ;  vd 
tods.  Sagrif  .can  mcdehng  gaps 
identficd 

ICME  development  plan  gnd  budget 
established.  ICME  modeling  system 
identified,  info  Mow  dis-grimmed,  I 
challenges  identified  and  integrated 
with  TRL  activities.  ICME  system 
development  team  membership  &  role: 
defined;  and  ICME  work  initiated  to 
meet  MRL-4  requitemehts.  ICME 
plan;  updated  bared  on  PIRT  results 
and  design  requirements. 

ICME  team  (with  MT  IPQT  and  TRL 
team)  scheduled  ICME  development 
experiments.  ICME  team  has  procured 
“system  engineering"  software  to 
conduct  analytical  DOE.  sensitivity, 
response  surface,  optimisation  and 
robust  manufacturing  analyses.  The 
CMC  system  satisfies  MRL -4 
requirements. 

iCMSE  "Digital  Thread"  data,  data 
base,  infrastructure,  and  technology:, 
that  support  ICME  development,  V4V. 
and  UC  analysis,  have  been  identified, 
designed  and  implemented.  ICSMlE 
subteam  has  conducted  a  Monte  C  arlo 
analysis  to  synthesize  MT  system 
variation  based  on  process  input 
variables  uneertaintyfvaiiation 

Development  priorities  and  new  CMC 
models  have  been  updated  baoed  on 
FMECA  results  to  achieve  IMRL-T 
requirements, 

ICME  :ubteam  ha;  developed  CMC 
oy; tern  that  satisfies  MRL-i 
requirement;. 

X3- 

Verification  & 
Validation  and 
ToolMatuiitj 
Level 

Initial  planning  V4V  Checklist 
completed  and  integrated  with  TRI 
plans  for  ICME  development  and 
associated  V4V.  Tool  Maturity  Levels 
(TML)  requirements  assessed  and 
coordinated  with  TRL  plans  to  guide 
planning  of  V4V  activities  and 
associated  budgetary  requirement ; 

TMLs  identified  for  models  4  tools . 
ICME  team  def  ined  and  scheduled  V4V 
experiments.  ICME  system  used  to- 
Identify  key  input  and  internal 
parameters  (influencing  simulation®  MT 
System  output  uncertainty.  ICME  tc  am 
[with  MT  IPDT  and  TRL  team)  have 
acquired  MT  development  data  for  U0 
of  hey  MT  process  input  variables. 
ICME  has  completed  MRL-i 
y&VfTML.  update;  and  requirement: 
have  been  met. 

ICMSE  subteam  has  designed 
verification  requirements  for  ICME 
system  models.  ICME  team  confirmed 
that  verification  requirement  :  for  ICME 
model;  have  been  achieved.  ICME  team 
has  designed  model  validation 
experiments.  ICME  team  has  designed 
validation  experiment:  for  the 
integrated  ICME  system. 

ICSME  subteam  ha:  competed  a  MFl 

T  update  of  the  Y4V  check!  | {As,  TML 
assessment,  VtV  Dashboard,  and 
concurrence  that  MRL-1  VI  tV 
requirement  have  been  met, 

X4  -  Diitical 
Experiments 
for  MT 
Development 

Motional  cipcrineMal  p  in  dt  *i»  :i 
neking  piogressiou  from  sail. 

;*pk  geometry  coupons  tlwowyh 
larger,  complex  jconutiv  sub-ekmeits 
r»d  M  soak  ankle ; 

High-level  experiment  plan  established 
and  budgetary  requirements 
determined  by  IPOT.  Process  feasibfcj 
demonstration  experiments  defined  an< 
successfully  conducted, 

Process  application  experiments 
defined  and  successfully  completed 
suitable  for  MRL4  level  of  application 
knowledge 

Criti  al  experiment  sfor  preices; 
demonstrations  and  success  ctrterim  for 
Specific  applications  defined  in  detail 
for  MRLb,  ?.  and  4  gate  requirement; 

Critical  experiments  for  representative 
stakvp  size  and  geometry  1‘eatme? 
juccessfullycompleted. 

Critical  e«p<rimeots  sucessfuly 
completed  on  representative  size  and 
geometry  artide(s). 

Cntical  txpenmcM:  inching  intended 
ROA  method:  :«ces:friy  completed 
on  dotaied  design-representative 
article 

X5  -  Ptocess 
Capability. 
Control,  and 
Quality 

Management 

ICME  Digital  Thread  enhanced  to 
Support  MT  quality  system. 

ICME  sensitivity  info  4  DC  data 
applied  to  guide  DC  data  acquisition 
plan.  ICME  4  SE  tools  applied  to 
improve  MT  system  dynamics, 
robustness,  and  control  system 
development.  Key  process  and  material 
parameter;  identified  for  active 
process  monitoring  and  control. 

Sensor  monitoring  and  key  data 
requirement;  identified, 

ICME  team  developed  an  a  pprooch 
and  software  to  implement  ftayesian 
updating  to  refine  assessment  of  QC. 
ICME,  and  quality  data,  Kej» 
processf  material  monitor  in-  y  4  control 
sensors  data  implemented .  Data  and 
process  output  integrated  -with  ICME 
models.  Process  Update  pit*: 
developed,  using  data  feed  bach  and 
ICME  models, 

Integration  of  hey  pioce::  data  a-d 
ICIVl  models  wim  UQ  and  variation 
a:;cssai<«ts  completed. level  of 
Siabstkai  Process  Cootrol  determined, 
and  requirement;  to  achieve  Six  Sigma 
cqurrakM  control  hue  bom  defined 

Sia-Sigma  or  equivalent  cot  fitted 
process  control  achieved  •  and 
ote  grated  with  ICME  model:  and  in  - 
proeess  marketing  apptoacie:  lot 
periodic  or  COM naous  feedback 
:u;tanmcr’,  andirprovemeM.  Froce:; 
lanabon  momtoved  and  contioledL 

X£- 

Manufacturing 

Management. 

Workforce 

Development. 

and 

Resources 

IPDfT  briefed mgt  on  SE  approach, 

TRL  integration,  4  expected 
acceleration,  quality  and  risk  reduction 
benefits.  IPDT  provided  mgt.  with  a 
plan  and  cost  analysis  for  MT  SE 
effort,  MT  mgt.  has  committed 
sufficient  funds  and  workforce 
resources  to  execute  systems 
engineering  accelef  Hioti  activities. 

IPOT  provided  manpower  plan  (in- 
housef out-sourced]  with  rationale  and 
costs,  Mgt,  committed  resources  (or 
hardware  4  software  for  ICME.  VtV, 
and  Digital  Thread  activities,  MT  IPDT 
has  summarized  workforce  need;  to 
support  MT  SE  development  4 
implementation. 

Workforce  development  activities  for 
training  in  ICME,  Uil,  V4V,  and  In- 
process  monitoring  methods  planned 
and  initiated. 

Figure  9:  Alignment  of  Proposed  SEMT  Thread  and  Six  Sub-threads  with  MRL  Gates 
Note  significant  activity  in  MRL  2  and  3  and  focus  prior  to  Milestone  B 


4.4  Recommended  Decision  Questions  for  When  to  Employ  SEMT 

The  proposed  activities  recommended  for  SEMT  represent  a  significant  commitment  of  time  and 
resources.  Consequently,  the  decision  to  employ  some  or  all  of  the  recommended  SEMT 
processes  should  be  carefully  considered.  The  perspective  on  the  need  for  such  activities  may 
differ,  depending  on  whether  the  perspective  is  that  of  the  technology  or  process  developers,  the 
product  or  system  chief  engineers,  or  the  program  management  (or  the  systems  acquisition 
management).  A  joint  assessment  and  consensus  decision  would  be  ideal,  although  an 
assessment  may  be  required  before  system  or  product  application  targets  can  be  well-defined.  A 
set  of  eight  (8)  questions  was  developed  to  assist  making  an  objective  decision  regarding 
implementation  (or  requirement)  of  the  SEMT  process.  The  questions  were  divided  into  two 
categories: 

•  Technology  and  Process  Development  Questions  -  these  were  intended  to  assist 
assessment  of  the  manufacturing  technology  or  process  development  itself,  and 

•  System  or  Product  Implementation  Questions  -  these  were  intended  to  assist  assessment 
of  the  impact  the  manufacturing  or  process  technology  could  have  on  the  intended 
application,  especially  if  MT  development  was  late  or  not  fully  successful  to  meet  system 
or  product  requirements. 

Since  the  eight  questions  were  necessarily  high-level,  a  set  of  sub-questions  was  also  developed. 
The  sub-questions  were  intended  to  clarify  intent  and  scope  of  the  Key  Questions,  and  provoke 
an  objective  assessment  of  the  need  for  the  SEMT  process.  The  questionnaire  is  presented  in 
Table  9,  below. 

The  recommended  approach  to  complete  the  questionnaire  is  to  review  and  answer  each  sub¬ 
question,  and  then  determine  the  overall  answer  to  the  corresponding  Key  Question.  Questions 
were  written  to  solicit  a  simple  Yes  or  No  response  to  each  item.  Selected  answer  boxes  have  a 
bold  outline  and  yellow  fill  color:  if  the  answers  to  one  or  more  sub-questions  are  in  such  a 
highlighted  box,  it  implies  that  the  Key  Question  answer  should  be  in  the  corresponding 
highlighted  box.  If  the  answers  to  any  of  the  Key  Questions  numbered  Ql,  Q3,  Q4,  Q6,  Q7,  or 
Q8  are  “Yes,  ”  the  recommendation  is  that  SEMT  processes  be  implemented.  Key  Questions  Q2 
and  Q5  may  indicate  risk  can  be  mitigated,  or  decision  to  employee  SEMT  delayed,  based  on 
other  applications  of  the  candidate  technology,  or  pending  better  definition  of  product  or  system 
application  requirements. 

The  questionnaire  was  developed  with  the  knowledge  (and  intent)  that  both  the  manufacturing 
technology  development  team  leaders,  and  the  product  or  system  application  engineers,  would 
likely  be  required  to  completely  answer  the  questions. 
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Table  9:  Proposed  Decision  Questionnaire  for  Use  of  SEMT  Process 

Utilization  of  Systems  Engineering  for  Manufacturing  Technology  (SEMT) 

Key  Questions  to  Assist  Implementation  Decision 


Technology  and  Process  Development  Questions 

1  Is  this  a  new  technology  or  manufacturing  process  for  aerospace  application? 

Does  MT  development  involve  a  new  class  of  material? 

Does  MT  development  represent  a  new  manufacturing  process? 

Does  MT  development  represent  more  than  a  derivative  or  upgrade  of  existing  technology? 

2  Has  the  manufacturing  technology  or  process  been  used  for  a  non-aerospace  application? 

Does  this  application  have  stringent  (aerospace-like)  quality,  performance,  and  durability  requirements? 
Is  there  available  literature  &  data  to  judge  the  suitability  of  the  MT  for  aerospace  applications? 

3  Have  significant  technical  risks  been  identified  for  the  manufacturing  technology  or  process? 

Has  a  risk  analysis  been  performed? 

Did  the  risk  analysis  identify  any  significant  risks? 

4  Will  new  infrastructure  or  supply  base  capability  be  required? 

Does  existing  supply  base  require  new  or  additional  capital  infrastructure? 

Are  significant  new  investments/development  required  for  production? 

Does  supply  base  require  development  of  new  or  significantly  different  skill  sets? 


Support 

Questions 

Key 

Questions 

Yes 

No 

Yes 

No 

System  or  Product  Implementation  Questions 

5  Has  a  system  application  of  the  technology  or  process  been  identiifed? 

Target  application  or  component/part  family  identified? 

Target  system  for  implementation  identified? 

6  Is  the  acquisition  program  accelerated  (per  Interim  DODI  5000.02)? 

Are  Technology  Development  and  EMD  activities  planned  to  overlap?  (Model  1) 

Are  Technology  Development  and  EMD  concurrent?  (Designated  as  an  Accelerated  Program,  Model  4) 


7  Does  the  technology/process  maturation  schedule  represent  a  risk  for  system  requirements? 

Is  the  technology  and/or  MT  expected  to  be  identified  as  a  "critical  technology?" 

Risk  that  technology/process  will  not  achieve  TRL/MRL  4  prior  to  Milestone  A? 

Risk  that  technology/process  will  not  achieve  TRL/MRL  6  prior  to  Milestone  B? 

Risk  that  technology/process  will  not  achieve  TRL/MRL  8  prior  to  Milestone  C? 

8  Is  the  Technology/Process  Strategic  to  the  System  or  Product? 

If  technology/process  is  late  to  schedule,  will  system  or  product  be  delayed? 

Will  an  acceptable  backup  for  the  technology/process  be  carried  as  risk  mitigation? 

If  technology/process  fails  to  mature,  will  system  or  product  arcitecture  have  to  change? 


HD 


IZZL 


DDL 


l=L 


Notes: 

Sub-questions  (indented)  are  intended  to  assist  answering  numbered  KeyQuestions.  Check  in  anybold  boxfor  sub-questions  infers  answer  to  KeyQuestion  is  also 
in  bold  box. 

2  Key  Technology  Questions:  Ifanswerto  Q1,  Q3,  orQ4  is  Yes,  Implement  SE4QRS  recommended  actions.  Maybe  mitigated  or  delayed  if  answer  to  Q2  is  Yes. 

Key  System  or  Product  Implemenation  Questions:  Ifanswerto  Q6,  Q7,  or  Q8  is  Yes,  Implement  SE4QRS  recommended  actions.  Maybe  mitigated  or  delayed  if 
answer  to  Q5  is  No. 


It  is  recommended  that  this  questionnaire  be  used  early  in  the  planning  of  an  MT  program.  The 
intent  is  to  provoke  an  objective  assessment  of  the  risks  posed  by  a  new  technology,  especially  to 
the  intended  system  or  product  application.  As  such,  any  manufacturing  technology  program 
would  likely  benefit,  whether  the  intended  application  is  a  “quick  reaction  system,”  a  designated 
“accelerated  acquisition  program,”  or  even  a  modification  to  an  existing  system  or  product. 
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5.0  CONCLUSIONS 


The  current  processes  and  guidelines  used  by  DoD  and  the  Air  Force  for  manufacturing 
technology  and  production  readiness  assessment  were  reviewed  under  this  study.  The  MRL  and 
MRA  processes,  considerations,  and  gate  exit  criteria  were  reviewed  in  detail.  Several  focus 
areas  were  identified  and  evaluated  as  candidate  methods  or  processes  that  could  accelerate 
maturation  of  new  materials  and  manufacturing  technologies,  and  reduce  the  risk  of  their 
implementation.  These  focus  areas  were  refined,  and  feedback  from  several  manufacturing 
technology  experts  in  AFRL  and  industry  was  solicited  and  incorporated. 

The  resulting  topics  comprise  a  suite  of  recommended  activities  that  collectively  were  termed 
“ Systems  Engineering  for  Manufacturing  Technology  (SEMT).  ”  The  individual  technical 
topics  that  were  identified,  as  well  as  supporting  infrastructure  and  management  activities,  were 
consolidated  into  six  defined  focus  areas.  These  are: 

1 .  Technical  Risk  and  Knowledge  Assessment :  including  expert  elicitation,  risk  assessment 
and  mitigation  methods,  and  non-advocate  review  processes 

2.  Integrated  Computational  Materials  Engineering  (ICME):  including  development  of  a 
strategy  for  comprehensive  computational  modeling  for  materials,  processing,  and 
properties,  including  identification  of  parameter  sensitivity  and  variation 

3.  Verification  &  Validation  and  Tool  Maturity  Level :  including  rigorous  model  V&Y, 
uncertainty  quantification,  and  assessment  of  analytical  or  computational  tool  maturity 
compared  to  application  requirements 

4.  Definition  and  Execution  of  Critical  Manufacturing  Technology  Experiments : 
including  a  strategy  for  integration  of  the  entire  planned  experimental  program  and  the 
definition  of  critical  demonstration  experiments 

5.  Process  Capability,  Control  and  Quality  Management  including  early  assessment  of 
key  process  parameters,  use  of  in-process  monitoring  and  control,  advanced  NDE 
methods,  and  Bayesian  or  other  continuous  updating  of  process  outputs  and  models 

6.  Manufacturing  Management,  Workforce  Development,  and  Resources :  including 
management  education  and  commitment  regarding  ICME  and  other  topic  areas  required 
for  SEMT,  workforce  development  in  SEMT  areas,  and  planning  and  commitment  of 
facilities  and  resources  to  accomplish  SEMT  activities  per  required  development  plans. 

The  current  MRL/MRA  process  is  well  defined  and  accepted;  it  is  also  comprehensive  in  both 
scope  and  application.  When  properly  practiced,  it  effectively  assesses  and  reduces 
implementation  risks  for  new  manufacturing  technologies.  The  proposed  SEMT  activities  are 
aligned  with  the  current  practice,  and  offer  the  potential  and  recommended  activities  to  enhance 
its  effectiveness,  accelerate  maturation  of  new  technologies,  and  reduce  their  risk  of 
implementation. 

How  to  effectively  implement  the  suite  of  activities  represented  by  SEMT  presents  a  significant 
challenge.  Conceptually,  the  six  focus  areas  were  viewed  as  possible  recommended  additions  to 
the  current  MRL/MRA  and  Manufacturing  Thread  construct.  Each  of  the  six  defined  focus  areas 
was  assessed  for  compatibility  with  the  current  MRL  gate  criteria  and  the  current  Manufacturing 
Threads.  Notional  activities  were  defined  for  each  focus  area,  and  aligned  with  appropriate  MRL 


45 

Distribution  Statement  A.  Approved  for  public  release  (PA);  distribution  unlimited. 


gates.  The  SEMT  activities  could  be  inserted  within  the  current  framework  in  a  piecemeal 
fashion,  inserted  as  an  additional  Manufacturing  Thread,  or  developed  as  a  separate  best  practice 
or  guideline,  compatible  with  the  current  process. 

The  proposed  activities  recommended  for  SEMT  represent  a  significant  commitment  of  time  and 
resources.  Consequently,  the  decision  to  employ  some  or  all  of  the  recommended  SEMT 
processes  should  be  carefully  considered,  and  should  involve  input  from  technology  or  process 
developers,  the  product  or  system  chief  engineers,  and  the  program  management  (or  the  systems 
acquisition  management).  A  one-page  questionnaire  was  developed  to  assist  making  an 
objective  decision  regarding  implementation  (or  requirement)  of  the  SEMT  process.  The 
questions  were  divided  into  two  categories:  one  representing  the  subject  technology  and  the  other 
category  representing  the  system  or  application  perspective  of  risk. 

The  key  conclusions  from  the  Study  Task  can  be  summarized  as  follows: 

1.  It  was  concluded  that  the  recommended  activities  within  the  six  defined  focus  areas 
offer  the  potential  to  greatly  accelerate  maturation  of  new  materials  and 
manufacturing  technologies,  and  significantly  decrease  their  risk  of  implementation. 

2.  One  key  conclusion  under  this  Study  Task  was  that  these  six  focus  areas  -  including 
both  strategies  and  methods  -  would  be  most  effective  if  employed  in  an  integrated, 
iterative,  and  collective  manner.  They  would  not  be  nearly  as  effective  if  applied 
individually  or  independently. 

3.  The  proposed  SEMT  activities  are  fully  aligned  and  consistent  with  intent  of  the 
current  MRL/MRA  process. 

4.  The  proposed  SEMT  activities  were  defined  in  a  manner  that  would  enable  inserting 
SEMT  as  an  additional  Manufacturing  Thread,  within  the  current  MRL/MRA 
process. 

5.  Alternatively,  it  was  concluded  that  the  SEMT  activities  could  be  developed  into  an 
engineering  or  program  best  practice  or  guideline,  and  implemented  within  the 
current  MRL/MRA  process  as  a  means  to  accelerate  designated  activities.  This 
approach  could  be  implemented  without  requiring  any  modification  to  the  current 
MRL/MRA  process.  This  approach  is  regarded^  the  preferred  path. 

6.  It  was  concluded  that  a  decision  process  for  when  to  implement  SEMT  was  required.  A 
one-page  questionnaire  was  developed  with  high  level  questions  intended  to  objectively 
assess  risk  of  the  technology  itself,  and  also  the  risk  posed  for  the  system  or  product 
application. 

The  most  significant  challenge  for  this  Study  Task  was  to  identify,  define,  and  recommend 
processes  and/or  methods  that  could  effectively  accelerate  maturation  of  manufacturing 
technologies,  and  to  provide  confidence  to  key  decision-makers  that  such  technologies  could  be 
successfully  implemented,  with  an  acceptable  level  of  risk.  A  set  of  activities  was  identified  and 
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evaluated,  with  potential  to  accomplish  these  goals.  These  focus  areas  have  been  aligned  with  the 
current  established  MRL  process,  and  recommendations  for  how  to  implement  SEMT  as  an 
either  as  an  optional  10th  Manufacturing  Thread,  or,  alternatively,  as  a  best  practice  or  guideline 
have  been  made. 
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6.0  RECOMMENDATIONS 


This  brief  “Phase  0”  Study  Task  was  intended  to  review  current  practices  and  background 
information,  identify  and  evaluate  promising  methods  or  processes  with  potential  to  significantly 
accelerate  maturation  of  manufacturing  technologies,  and  to  provide  confidence  to  key  decision¬ 
makers  that  such  technologies  could  be  successfully  implemented  with  an  acceptable  level  of 
risk.  A  set  of  activities  defined  as  “ Systems  Engineering  for  Manufacturing  Technology 
(SEMT)  ”  was  been  identified  and  evaluated,  The  six  focus  areas  comprising  SEMT  were  aligned 
with  the  current  established  MRL  process,  and  recommendations  for  how  to  incorporate  them 
have  been  made.  Although  several  manufacturing  technology  experts  did  review  the  concepts 
and  proposed  focus  areas  during  the  course  of  this  study  task,  the  results  in  this  final  report  have 
not  yet  been  comprehensively  reviewed. 

Consequently,  the  near-term  focus  for  future  activities  is  on  presenting  the  SEMT  concept  to 
interested  parties  in  government  and  industry,  soliciting  feedback  and  constructive  criticism,  and 
incorporating  recommended  revisions.  The  objective  is  to  solicit  not  only  feedback,  but 
hopefully  support  and  advocacy  for  implementation  of  SEMT  in  some  manner,  either  as  part  of 
the  established  MRL/MRA  process,  or  as  Recommended  Best  Practice  or  engineering  or 
program  work  instruction.  This  would  ideally  involve  at  least  the  following  recommended  steps: 

1.  Presentation  and  reviews  with  AFRL/RXM,  MAI,  and  possibly  others  are 
recommended.  These  reviews  could  be  constructed  and  abbreviated  from  the  final 
summary  presentation,  which  was  delivered  by  the  authors  to  AFRL/RXM  on  23  July 
2014. 

2.  Distribution  of  the  Study  Task  Final  Report  to  interested  parties  for  review  and 
feedback  is  recommended. 

3.  Review  with  the  JDMTP  committee,  as  the  owners  of  the  MRL/MRA  process,  is 
recommended.  This  is  regarded  important,  whether  SEMT  is  advocated  as  an 
additional  Manufacturing  Thread  or,  alternatively,  as  a  best  practice  or  guideline 
process. 

Provided  that  response  regarding  proposed  SEMT  activities  is  positive  and  supportive,  the 
following  actions  are  also  recommended: 

4.  Development  of  some  plans  specifically  for  SEMT  revision,  beta-testing  and  an 
initial  application  exercise  under  an  actual  manufacturing  technology  program 
is  recommended.  It  may  be  advisable  to  select  an  existing  candidate  MT  program, 
where  the  planning  and  strategy  aspects  of  SEMT  could  be  exercised  independent  of 
the  existing  program  plan  and  statement  of  work.  Ideally,  a  program  could  be  selected 
-  existing  or  new  start  -  where  SEMT  would  be  appropriate  and  could  be  fully 
exercised. 

5.  It  is  recommended  that  an  approach  or  plan  identifying  the  implementation  path 
and  actions  required.  This  may  be  to  define  SEMT  in  the  form  of  engineering  or 
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program  instruction,  a  best  practice  or  guideline,  or  an  ad-hoc  addition  to  the 
MRL/MRA  process. 

6.  Community  reviews  and  focus  sessions  are  recommended,  via  Technology 
Interchange  Meetings  or  workshops  to  engage  Original  Equipment 
Manufacturers,  suppliers,  and  government  experts. 

7.  An  effort  to  produce  a  final  (Version  1)  SEMT  Guideline  suitable  for 
implementation.  The  form  this  would  take  would  be  dependent  upon  outcome  of 
Recommendation  5,  above. 

Although  no  specific  time-line  is  recommended  for  the  above  items,  it  is  recommended  that 
Items  1,  2,  and  3  be  addressed  in  calendar  year  2014.  Items  4,  5,  and  6  are  contingent  on  outcome 
of  Items  1,  2,  and  3,  and  would  follow,  probably  beginning  in  early  2015. 

Overall,  the  proposed  SEMT  manufacturing  thread,  with  its  six  focus  areas  defined  as  sub¬ 
threads,  represents  a  comprehensive  and  integrated  set  of  activities.  It  is  believed  that  SEMT 
would  be  a  very  useful  addition,  revision,  or  supplement  to  the  current  MRL/MRA  process.  In 
addition,  it  is  believed  that  SEMT  may  prove  to  be  a  very  useful  set  of  activities  in  support  of 
three  significant  AFRL/RXM  focus  areas:  Moving  Manufacturing  Left  (MML),  Agile 
Manufacturing,  and  Digital  Thread. 
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APPENDIX:  EXPERT  ELICITATION 

Successful  expert  elicitation  (EE),  whether  using  Phenomena  Identification  and  Ranking 
Technique  (PIRT),  Failure  Mode,  Effects,  and  Criticality  Assessment  (FMECA),  or  other 
methods,  depends  foremost  on  having  a  high  level  of  commitment  to  EE  within  the  MT 
development  team,  particularly  among  the  IDPT  leadership.  Unenthusiastic  effort  in  composing, 
instructing,  and/or  informing  the  EE  team,  will  weaken  EE  team  performance  and  lessen  the 
benefits  of  the  EE  team’s  work. 

As  a  prelude  to  selection  of  EE  panel  members,  the  IPDT  can  conduct  preliminary  PIRT  and  risk 
assessment  exercises  to  identify  those  elements  of  the  MT  development  that  are  most  important, 
least  well  understood,  or  subject  to  risk;  and  also  list  relevant  successor  technologies.  Armed 
with  this  information,  the  IPDT  can  better  focus  the  EE  panel  selection  process  on  key  categories 
of  expertise  of  greatest  importance  to  the  MT  development  program;  and  gain  clues  as  to  where 
the  IPDT  can  find  the  right  experts,  whether  from  within  the  resident  engineering  staff, 
academia,  government  laboratories,  or  other  industries  involved  in  related  MT  sub-processes. 

First,  the  IPDT  needs  to  identify  a  group  of  willing  experts  that  collectively  span  the  breadth  and 
depth  of  the  lesser-known  process  steps  of  the  subject  manufacturing  technology.  These  team 
members  should  be  independent,  meaning  they  should  not  have  a  vested  interest  in  the  EE 
outcome  or  the  success  of  the  MT  program  [A-l],  Although  assembling  such  a  team  is  often  not 
easy,  the  quality  of  the  EE  team  findings  and  recommendation  depend  on  finding  highly 
experienced  and  knowledgeable  experts,  preferably  individual  unafraid  to  challenge  the 
manufacturing  technology  under  consideration  or  the  technical  viewpoints  and  analysis  of  the 
IPDT  membership.  Indeed,  the  IPDT  leadership  should  encourage  a  skeptical  outlook  on  the 
part  of  EE  team  members. 

Second,  the  IPDT  needs  to  provide  the  EE  team  with  explicit  instructions  so  that  it  clearly 
understands  the  scope  and  objectives  of  the  exercise,  along  with  instructions  for  the  EE 
process(es)  that  will  be  used  in  the  investigation.  The  EE  team  should  not  be  allowed  to  either 
assume  or  define  the  scope  or  objectives  of  the  EE  exercise  without  consulting  with  the  IPDT 
and  gaining  approval  for  any  change  that  the  EE  team  might  recommend  [A-l]. 

Third,  the  IPDT  needs  to  collect  and  disseminate  information  to  the  EE  team  to  provide  MT 
technology  background,  development  status,  and  specific  technical  perspectives  of  IPDT 
members.  This  input  would  include  the  following  types  of  information  [A-l]: 

•  An  overview  of  the  MT  development  program,  requirements,  perceived  challenges,  and 
expected  benefits; 

•  Historical  information  describing  related  predecessor  or  constituent  processing 
technologies,  along  with  their  strengths  and  weaknesses; 

•  A  summary  of  the  target  manufacturing  technology  that  includes  a  flow  diagram  of 
process  steps,  description  of  intended  processing  equipment,  and  identification  of  new  or 
unfamiliar  steps; 

•  Thorough  documentation  describing  state-of-development  for  the  manufacturing 
technology  including  important  operational  experience,  data,  and  suspected  shortcomings 
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•  Information  relative  to  key  process  phenomena,  ICMSE  capability,  process  sensing,  in- 
process  NDE,  and  testing  of  intermediate  product; 

•  The  results  from  any  risk  analysis  and/or  PIRT/FMECA  exercises  conducted  by  the  IPDT 
team;  and 

•  Candid  discussion  about  the  spectrum  of  issues,  questions,  risks,  and  other  perspectives 
within  the  MT  IPDT  that  collectively  should  explain  why  the  EE  team  was  empanelled. 

In  some  regards  the  expert  elicitation  team  should  function  much  like  a  “Red  Team”  review 
panel,  albeit  one  that  exercises  a  well-defined  EE  process.  And  much  like  the  effectiveness  of 
“Red  Team”  reviews,  the  success  of  expert  elicitation  depends  very  strongly  on  whether  IPDT 
leadership  believe  the  process  is  a  meaningful  resource  to  enhance  their  MT  development 
progress  -  as  opposed  to  simply  being  a  process  needed  to  comply  with  a  contract  checklist 
requirement  or  a  preference  of  the  program  sponsor. 

Phenomena  Identification  and  Ranking  Technique  (PIRT) 

The  US  Nuclear  Regulatory  Commission  developed  PIRT4  in  1989  as  part  of  their  “Code 
Scaling,  and  Applicability”  effort  [A- 1  ] .  The  premise  of  PIRT  is  to  gather  information  about 
phenomena  governing  subject  process/equipment  from  a  panel  of  experts.  The  EE  team  ranks 
the  importance  and  state-of-process  knowledge  for  the  phenomena  to  support  program  decision 
making,  which  includes  identification  of  key  unknowns  and  help  to  define  research  that  will  fill 
important  knowledge  gaps  [A-2],  In  execution,  this  strategy  identifies  and  ranks  phenomena 
based  on  their  effect  upon  key  manufacturing  and  product  requirements,  referred  to  as  the 
figures-of-merit  (FoM).  Application  of  PIRT  should  adopt  a  hierarchal  approach  that  starts  with 
lower-level  process  sub-steps  and  proceeds  upwards  towards  the  complete  system.  The  PIRT 
strategy  is  executed  in  four  successive  steps,  including  the  PIRT  scoring  table  shown  in  Figure 
A-l: 

1.  Identify  phenomena  that  control  process  outcomes.  Phenomena  can  be:  characteristics  of 
input  materials,  design  elements  of  manufacturing  equipment,  a  process  parameter 
window,  a  control  algorithm,  a  mechanism  within  a  ICMSE  model,  or  anything  else  that 
might  influence  the  FoMs; 

2.  Estimate  the  level  of  importance  for  each  phenomena  as  it  affects  each  FoM,  using  the 
rating  system  outlined  below; 

3.  Estimate  the  level  of  knowledge  uncertainty  for  each  the  phenomena  using  the  rating 
system  below; 

4.  Collect,  enumerate,  and  tabulate  the  ratings  from  the  EE  panel  members  and  calculate  the 
overall  importance  and  uncertainty  rankings  using  the  collective  voting  results  from  the 
team. 

Importance  Level  (how  much  do  the  Phenomena  affect  the  FoM): 

1 .  High  (H)  implies  phenomenon  (or  parameter,  model . . .)  has  a  controlling  impact  on  a 
given  FoM. 


4  PIRT  is  a  widely  known  process  that  is  well  documented  in  the  literature,  much  of  which  is  available  on  the  web.  This  description 
of  PIRT  attempts  only  to  capture  the  highlights  and  provide  a  simple  representation  of  supporting  charts  used  for  analysis. 
Interested  readers  are  encouraged  to  read  the  documents  referenced  herein. 
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2.  Medium  (M)  implies  phenomenon  has  moderate  impact  on  a  FoM  &  only  moderate 
accuracy  is  required  for  manufacturing  parameters,  measurements,  controls,  or  associated 
models. 

3.  Low  (L)  implies  the  phenomenon  has  very  little  or  no  impact  on  the  FoM. 

Knowledge  Level  {how  well  understood  is  the  phenomenon,  parameter  or  model): 

1 .  Known  (K)  implies  knowing  at  least  75%  of  what  we  could  expect  to  be  known. 

2.  Partially  Known  (P)  implies  that  25-75%  of  needed  knowledge  is  established. 

3.  Unknown  (U)  implies  that  less  than  25%  of  needed  knowledge  is  established. 

After  the  EE  panel  has  completed  the  PIRT  ranking  table  (Figure  A-l),  the  panel  and  the  MT 
IPDT  can  use  the  results  to  help  focus  the  development  program.  The  teams  can  identify  those 
phenomena  upon  which  research  is  needed  by  plotting  knowledge  uncertainty  versus  phenomena 
importance,  as  depicted  in  Figure  A-2. 


Phenomena 

Importance 

Knowledge 

Uncertainty 

Subcategory 

(within  the 

phenomena  hierarchy) 
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P5 

P6 
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P7 

P8 

P9 
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Pll 
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NOTES:  (1)  NE1  thru  NE6  represent  the  number  of  experts  scoring  these  categories 

(2)  Importance  Ratio  (IR)  =  (NE1  +0.5*NE2)  /  (NE1  +  NE2  +  NE3) 

(3)  Uncertainty  Ratio  (UR)  =  (NE4  +0.5*NE5)  /  (NE4  +  NE5  +  NE6) 

(4)  The  Uncertain  Scale  (defined  in  note  43)  is  the  inverse  of  that  used  by  Diamond  (2) 


Figure  A-l:  The  PIRT  ranking  table  enumerates  the  number  of  PIRT  panel  members  who 
rate  importance  in  the  H/M/L  columns  and  knowledge  uncertainty  in  the  U/P/K  columns. 
Formulas  shown  in  the  notes  are  then  applied  to  calculate  the  “importance  ranking”  (IR) 

and  the  “uncertainty  ranking”  (UR) 
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Figure  A-2:  The  PIRT  ranking  table  plots  “importance  ranking”  (IR)  and  “uncertainty 
ranking”  (UR)  for  each  phenomena  to  provide  a  graphical  view  of  where  research  is 

needed 


This  chart,  when  compared  to  initial  IPDT  risk  assessments,  can  highlight  those  phenomena  and 
their  associated  process  steps  that  may  pose  potential  locations  for  Unk-Unk  risk.  The  PIRT 
process  is  particularly  valuable  during  early  stages  of  MT  development  (or  even  during  pre¬ 
program  planning),  whether  conducted  by  the  IPDT  alone  or  in  conjunction  with  an  independent 
EE  panel. 

This  section  attempts  to  provide  an  overview  of  PIRT;  indeed  there  are  many  exhaustive  reports 
describing  the  process,  some  of  which  are  listed  in  the  references.  Foremost,  however,  IPDT 
teams  need  to  understand  the  underlying  PIRT  concept,  rationale,  and  general  approach,  and  then 
tailor  a  PIRT  exercise  to  their  programs  needs.  As  espoused  before,  early  application  of  EE  and 
the  inclusion  of  independent  experts  in  the  PIRT  process  can  significantly  enhance  program 
success  by  reducing  the  incidence  of  unexpected  surprises  that  may  require  program  redirection, 
added  cost,  and  possibly  even  abandonment. 

Failure  Mode.  Effects,  and  Criticality  Assessment  (FMECA) 

NASA  developed  the  FMECA  process  to  improve  and  verify  the  reliability  of  space  program 
hardware.  The  process  is  documented  in  MIL-STD-1629  and  is  extensively  discussed  in  a  1993 
DLA/DTIC  report  [A-3],  NASA’s  premise  for  FMECA  was  to  gather  information  from  experts 
who  are  charged  to  identify  potential  failure  modes  for  a  system  or  product,  assess  their  effect, 
and  then  determine  the  criticality  of  the  failure  in  support  of  program  decision-making.  While 
developed  for  space  program  hardware,  FMECA  has  been  applied  to  a  wide  variety  of  other 
systems  and  is  certainly  applicable  to  manufacturing  development  technologies  -  including  both 
the  manufacturing  process  and  the  resulting  manufactured  product.  It’s  also  important  to  note 
that  the  IPDT  and  EE  panel  can  flexibly  define  the  word  “failure”  to  conform  to  the  specific 
needs  of  the  FMECA  assessment.  In  other  words,  failure  might  mean  might  failure  of  process 
sensor,  but  it  could  also  refer  to  excessive  variation  for  a  critical-to-quality  (CTQ)  product 
attribute,  etc.  It’s  also  useful  to  contrast  FMECA  with  the  PIRT  process.  Whereas,  PIRT  strives 
to  identify  key  phenomena  and  then  deduce  their  effects  and  associated  knowledge  gaps; 


55 

Distribution  Statement  A.  Approved  for  public  release  (PA);  distribution  unlimited. 


FMECA  attempts  to  evaluate  hypothetical  failure  modes  (what  might  go  wrong)  and  then 
evaluate  their  likelihood  and  impact.  As  such,  FMECA  represents  a  much  different,  but 
complementary,  thought  process  from  that  used  by  PIRT. 

The  more  qualitative,  functional  FMECA  approach  is  likely  most  germane  to  early 
manufacturing  technology  development.  This  approach  follows  the  following  steps: 

1.  Identifying  potential,  signification  MT  process  mishaps,  disruptions,  or  failures  relative 
either  to  MT  process  flow  or  product  attributes.  Failure  might  be:  formation  of  a  material 
defect,  excessive  residual  stress,  properties  outside  specification  limits,  inadequate  NDE 
probability  of  detection,  process  control  instability,  or  excessive  variability  of  process 
parameters,  etc. 

2.  Determining  the  severity  of  the  “failure”  upon  associated  FoM’s; 

3.  Estimating  the  probability  of  “failure;  and 

4.  Estimating  the  joint  probability  that  the  effect  or  “failure”  will  be  detected  using  process 
diagnostics  or  NDE. 

5.  Finally,  the  assessment  team  should  aggregate  and  interpret  results  based  on  the  FMECA 
chart,  shown  in  Figure  A-3. 

Given  the  likely  lack  of  detailed  process  data,  for  steps  2  through  4  early  in  the  program,  the 
IPDT  team  should  rank  the  items  #2  through  #4  qualitatively  based  on  team  judgment  using  a  1- 
5  scoring  system.  There  are  a  number  of  FMECA  resources  available  in  the  literature  that  can 
provide  additional  insight  on  how  to  best  execute  this  process  for  a  given  MT  development 
program.  Also,  prior  to  committing  to  an  EE  exercise,  the  team  must  assess  whether  the  MT 
development  represents  a  MT  refinement  or  a  greater  leap  forward.  It  must  also  judge  the  IPDT 
level  of  knowledge  about  new  manufacturing  technology  domain  that  will  be  explored  during  the 
manufacturing  development  program.  These  judgments  are  critical  to  program  success;  and 
potential  risks  can  be  lessened  when  the  team  acknowledges  program  deficiencies  and  accepts 
that  there  may  be  experts  in  seemingly  unrelated  fields  that  can  provide  very  useful  guidance. 
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Notes:  (1)  Entries  under  the  "Potential  Effect"  column  should  specify  the 
requirement  (FoM)  effected; 

(2)  Numeric  values  should  be  used  to  score  columns  A,  B,  C  (e.g.,  1-10) 

(3)  Risk  Priority  Number:  D  =  A  *  (1-B)  *  (1-C) 

Figure  A-3:  Typical  chart  used  to  tabulate  FMECA  results  and  then  calculate  analysis 
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LIST  OF  SYMBOLS,  ABBREVIATIONS,  AND  ACRONYMS 

Air  Force  Research  Laboratories 
Defense  Acquisition  Guidebook 
Defense  Research  Projects  Agency 
Design  for  experiments 
Department  of  Defense 
Department  of  Defense  Instruction 
Design  of  experiments 
Expert  elicitation 

Engineering  and  Manufacturing  Development 
Foundational  Engineering  Problem 
Figure  of  merit 
Growth  rate 

Government  Accountability  Office 
Hot-isostatic-press 

Integrated  Computational  Materials  Engineering 

Integrated  Computational  Materials  Science  and  Engineering 

Integrated  Product  Development  (or  Deployment)  Team 

In-process  quality  assurance  (trademark  of  Beyond  Six  Sigma) 

Joint  Defense  Manufacturing  Technology  Panel 

Key  process  characteristic 

Low  Rate  Initial  Production 

Metals  Affordability  Initiative 

Manufacturing  Technology 

Materials  &  Processes 

Manufacturing  Readiness  Assessment 

Manufacturing  Readiness  Level 

Manufacturing  Technology 

Non-advocate  review 

Non-destructive  evaluation 

Phenomena  Identification  and  Ranking  Technique 

Pratt  &  Whitney 

Prior  particle  boundary 

Quality  control 

Thermal  gradient 

Systems  Engineering 

Systems  Engineering  for  Manufacturing  Technology 

Acronym  for  the  Manufacturing  Technology  Directorate  of  AFRL 

Statement  of  Work 

Southwest  Research  Institute 

Tool  Maturity  Level 

Technology  Readiness  Assessment 

Technology  Readiness  Level 

Uncertainty  quantification 

Verification  &  validation 
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